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PREFACE 



‘^Programed Instruction in Occupational Education' ' is the 
result of interest shown in the previous publication, **Using Pro- 
gramed Instructions With and Without Self-Instructional Practice 
to Teach Psychomotor Skills" by the same author. 

The bulletin is intended as a guide to occupational teachers, 
supervisors, and administrators in planning, developing, and using 
various types of programed instruction as a guide in the learning 
process. It is desired that the information presented herein will 
result in units of instruction where students and teachers may 
benefit from such facilities useful in learning new skilla to increase 
cognitive faculties in specific occupational areas. 

The writer wishes to acknowledge the supervisory assistance 
of Professor Joe P. Bail, Chairman of the Agricultural Education 
Division at Cornell, who served as Project Director, and to Pro- 
fessor Frederick K. T. Tom, of the Agricultural Education Division 
at Cornell, who served as Project Director during the planning 
stages. 

Similarly, gratitude is due Lyle Wicks, Instructional Materials 
Specialist, Division of Agricultural Education, for his many helpful 
suggestions and his assistance with the production of this publica- 
tion. 

Appreciation is expressed to Dr. Alan Robertson, formerly 
Chief of the Bureau of Occupational Education Research, who was 
responsible for initiating the project, and to Charles Meislin, an 
Associate in the Bureau, who served as project officer. Thanks 
are also due Dr. Louis Cohen, presently Chief of the Bureau, for 
his assistance with the final stages. 
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CHAPTER I 



INTRODUCTION 



In an age where the acquisition of knowl- 
edge is the way of life for most people, pro- 
gramed instruction offers much: individual- 
ized rate of instruction, faster and more 
effective learning, immediate confirmation 
of student responses. Many students, espe- 
cially those with low ability, experience for 
the first time a new feeling of success and 
confidence in their ability to learn. By means 
of carefully constructed sequences, the pro- 
gram takes each learner from what he knows 
to what we want him to know. 

Today professional literature abounds 
with reports attesting to the fact that pro- 
gramed instruction has proved itself in 
industry, in the colleges, and in the armed 
forces. It is high time that programed in- 
struction redeemed its promises in our 
schools. What can be done more effectively 
and more efficiently with this method should 
be done! 

Purpose and Scope 

The purpose of this publication is to pre- 
sent an up-to-date report on the status of 
programed instruction and the potential it 
holds for our schools. Most educators have 
not yet been convinced of the value and 
effectiveness of programed instruction, nor 
are they aware tnat it is a direct effort to 
implement modem le arning theories and the 
results of current research by building them 
int o a more effective, more scientific tech- 
nology of teaching and learning. 

The emphasis here is on the application 
of programed instmction to occupational edu- 
cation, but much of what is said is also appli- 
cable to general education. 

An explanation of what programed in- 
struction is, and an overview of supporting 
psychological principles, research findings, 
some controversies, some advantages, and 
some limitations of programed instruction 
are presented in Chapter 2. Chapter 3 gives 
practical and useful information on the selec- 
tion and use of programed materials, and the 



fourth chapter cites numerous case studies 
taken from the experiences of schools and 
industry where programed instruction has 
been used in occupational education. 

Three of the four appendices will be of 
special interest to teachers, administrators, 
and others v^o want to: (1) leam what mate- 
rials are available commercially, and their 
sources (Appendix A), (2) read further about 
programed instruction (Appendix B), and (3) 
learn more about recommended procedures 
for selecting programed materials (Appendix 
C). 

Historical Sketch 

No single person can claim responsibility 
for the introduction of programed instruction. 

Instead, three persons - Sidney L. Pressey, 
B. F. Skinner, and Norman Crowder - are 
recognized as the outstanding contributors to 
its development. Teaching machines were 
first discussed by Dr. Pressey at an Amer- 
ican Psychological Association meeting held 
in 1924. Pressey^s approach is based on 
recognition of a correct response. 

With the exception of research scientists 
and a few imiversity educators, very little 
attention was paid these devices until the 
1950^s. A paper on conditioned learning by 
Harvard's Dr. Skinner, read before a con- 
ference of psychologists in 1954, is believed 
to have created a stir far beyond his im- 
mediate audience. Skinner, a well-known 
experimental psychologist, began to publish 
articles about the teaching machines he and 
his colleagues had developed. Much of his 
work was based on the principle of recall of 
information, a major feature of linear pro- 
graming. 

Because of serious teacher shortages and 
the need for new and improved methods of 
instruction, his writings were widely dis- 
cussed and stimulated many people. One of 
the more notable of these was Norman Crowd- 
er who initiated and carried out extensive 
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work with the branching technique, known as 
intrinsic programing. During the last few 
years so many researchers have turned their 
attention to programed instruction that pro- 
fessional literature in education has been 
flooded with articles and research repoits 
on it. 

Who is Using Programed Instruction? 



Unfortunately, there is no up-to-date in- 
formation on the percentage of schools in the 
United States that is actually usingprograms 
published commercially or produced locally. 
Kenneth Komoski, director of the Center for 
Programed Instruction, Columbia University, 
stated in 1965 that a conservative estimate 
would probably not go beyond 10 per cent, 
but that an estimate of 20 per cent might not 
be far from the mark either. He also re- 
ported a great deal of international activity 
in programed instruction. Including work by 
educational ministries, other government 
agencies, universities, and industries, in 
1965 between twenty and thirty countries 
were involved in researching, developing, 
and/or using programed instruction. Its 
proliferation in many directions is such that 
it must surely stand as one of the most 
rapidly developing and diffusing educational 
innovations of all time. 

Far more startling is the use of pro- 
gramed instr*iction by industry. All types of 
organizations - from the industiial giants 
(Gulf, Ford, General Motors, DuPont, Amer- 
ican Telephone and Telegraph, American Oil, 
U. S. Steel, IBM) down to local businesses 
(auto dealers, supermarkets, banks, tool and 
die shops, and department stores) - are 
using programed instruction wherever there 
is a training job to be done. 

Illustrative of the extensive use of pro- 
gramed instruction by companies are two 
reports, one by the DuPont Company and the 
other by the Argyle Publishing Corporation. 
According to Dr. Arthur Santora, head of 
DuPont* s Industrial Training Service, what 
DuPont did not intend to do, at the time they 
beg^ working with programed instruction in 
1959, was to get into the teaching business. 
But the same factor that got them started - 
a lack of commercially available courses in 
the desired skills - created a demand for 
their courses outside the company. Above 



and beyond their own internal demand of 
55,000 course units in the last five years, 
there has been an outside demand for an 
equal number of course units in less than 
two years! Courses have been supplied to 
1,200 other companies, many government 
a. id state agencies including the Bureau of 
Prisons and the Job Corps, hospitals, insti- 
tutions, and vocational schools. Similarly, 
an Argyle sales brochure lists 132 major 
organizations including the Army, Navy, and 
Air Force and states that these are a few of 
the more than 9,000 companies now using 
Argyle programed instruction. 

Education, however, is not keeping pace 
with industry. A 1966 National Education 
Association study of programed instruction 
in large school systems in the United States 
reports the grades and number of programs 
used in 126 school systems. A glance at 
Figure 1 provides an idea of where most 
programed instruction is presently being 
used in our elementary and secondary 
schools. 

The largest percentage of use has been at 
the secondary level, with the ninth grade being 
the level at which the most extensive use 
occurred. 

The Need and the Responsibility 

Modem society is characterized byahi^ 
degree of technological innovation and sophis- 
tication. Joggled by a technology that changes 
faster than feminine fashions, industry is 
devoting ever more attention to training, re- 
training, and upgrading the abilities of its 
workers. Correspondingly, we mi^t expect 
that through research and development edu- 
cation would also have turned to technology 
for assistance in meeting its increasingly 
important objectives, but actually education 
has not been characterized by technological 
innovation. Only recently have perceptive and 
forward-looking educators turned to tech- 
nology as a means to improve education. 
Education has witnessed a technological lag 
rather than a technological revolution. 

Are yesterday* s methods and techniques 
capable of meeting today* s needs? In the not 
too distant past, a man could learn a trade 
and look forward to practicing it the rest of 
his life without having to learn much of any- 
thing new. Today vocations change in many 
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Figure 1. Grades in which programed instruction 
was used in 126 large school systems 
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important wajns during each man^s productive 
years. Every man must be prepared to 
learn and relearn a living throu^out his 
life. 

Educators have long idealized but never 
realized the educational goal of adapting 
instruction to the ability of the individual 
student. We are constantly pitching our 
classroom instruction to the level of the 
mythical average student while those of 
hi^er ability are bored, and those of lower 
ability, baffled. Can we allow this injustice 
to so many students to continue? 

The findings of multicounty surveys in 
New York State indicate another definite 
need to expand and improve occupational 
education in New York State. Three of these 
findings, as presented in Education for Occu- 
pations , are as follows: 

1. Statewide trends indicate a continuing 
need for development of hi^er skill 
and technological knowledge compe- 
tencies among the work force. 

2. Statements by employers and \mions 
in the area studies certify that persons 
with vocational-technical skills needed 
in the work force are in short supply. 



3. The multiple area studies confirm that 
many present school programs are 
inadequate in providing needed salable 
skillsT 

A representative of many industries, the 
National Association of Manufacturers, sup- 
ports these findings in a case study which 
reports that many jobs continue to go begging 
because thousands of untrained or untrainable 
imemployed lack necessary basic educational 
skills. 

What is our responsibility as educators? 
A statement printed in the NEA^s January 
1963 issue of A udiovisual Instruction relates 
our responsitiliTity so well that It is repeated 
llOI'O 

Education is a matter of individual 
human growth and development; there- 
fore technological methodology must 
be introduced with care. Our primary 
concern is and must be the indiviaual 
human personality - the nurturing of 
social and moral values, and, perhaps 
more vo the point in our case, the 
development of rational thinking, of 
intellectual competence, of respon- 
sible action, and of productive ability. 
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Such development involves the trans- 
mitting (teaching) and mastering 
(learning) of a great deal of informa- 
tion and many complex skills. 

The responsibility is clear and the need 
is evident. With full realization of the task 
to be accomplished, and the ever increasing 
shortage of qualified teachers, educators 
must turn to technology. Again to quote 
Audiovisual Instruction , ** A technological 
leap forward is required in education. 

Traditional Instruction versus an Instruc- 
tional System 

The very common direct teacher-pupil 
relationship, the basis of traditional educa- 
tion, is illustrated in Figure 2, taken from 
the NEA's January 1963 issue of Audiovisual 
Instruction. 



This approach presents a scientifically de- 
veloped arrangement of instructors, mate- 
rials, and technological media for providing 
optimum learning with a minimum of routine 
personal involvement by the teacher. The 
result, as depicted in Figure 3, is a carefully 
planned systemx consisting of subject matter, 
procedures, and media coordinated in a pro- 
gram-unit design which is directed toward 
specific behavioral objectives. 

The instructional system approach pro- 
vides three alternate instructional routes and 
allows the teacher to select the most effective 
one for each learning situation. The teacher 
xnay present new material himself, select 
various media to help him with the presenta- 
tion, or utilize media alone to make the pre- 
sentation. 

Another feature of the instruction system 
approach is that it provides for regular feed- 



Figure 2. Traditional Instruction 




Although printed materials, chalk, and a 
few other devices came into play, there^ is 
little real technology involved in this in- 
structional organization. 

The new approach to a systems concept 
of instruction requires our earnest attention. 



back and evaluation at all stages of the in- 
structional process . Such feedback and eval- 
uation, properly utilized, provides valuable 
information for revising objectives, altering 
curriculum content, and selecting the best 
methods. 



Figure 3. Instructional System 



























Teacher 


















Alone 












Content 

and 

Method 










Objectives 


- 






Teacher 

with 

Media 






Pupil 






u6Cisions 




















Media 








♦ 










Alone 






1 












1 




. _ __ L. _ 




1 






1 



Feedback & Evaluation 



o 



4 



The NEA article summarizcjs the function 
of educational media as twofold. 

The first function of technological 
media is to supplement the teacher 
through increasing his effectiveness in 
the classroom. Educational media are 
both tools for teaching and avenues for 
learning, and their function is to serve 
these two processes by enhancing 
clarity in communication, diversity in 
method, and forcefulness in appeal. 
Except for the teacher, these media 
will determine more than anything else 
the quality of our educational effort. 

The second function of media is 
one in which the media alone may 
present and, in a sense, teach certain 
content to pupils. Here, the teacher 
determines objectives, selects meth- 
ods and content, and evaluates the final 



learning outcomes. The presentation 
of information, and even the direction 
of routine pupil activities, may be 
turned over to such new media as 
programed learning materials, tele- 
vision, or motion pictures. Function 
No. 2, then, is to enhance overall pro- 
ductivity throu^ instructional media 
and systems which do not depend upon 
the teacher for routine execution of 
many instructional processes or for 
clerical-mechanical chores. 

As the training director of a large New 
York City bank points out, “No one training 
method meets all requirements. A profit- 
oriented company or an educational system 
must provide the most effective training and 
education, using the most up-to-date and 
effective methods, and at the least cost con- 
sistent with good judgment.” 



CHAPTER II 

PROGRAMED INSTRUCTION - AN EXPLANATION AND OVERVIEW 
OF A NEW TECHNOLOGY OF TEACHING 



What is Programed Instruction 

Programed instruction is one of several 
relatively new technological innovations in 
education. It is a process or technique for 
the design and development of self-instruc- 
tional materials. A program of instruction 
is simply an instructional unit developed by 
means of this process. The process con- 
tains these essential steps: (1) establish- 
ment of specific behavorial objectives, (2) 
analysis of instructional content, (3) pro- 
gram production, (4) preliminary testing, (5) 
revision, (6) field testing, (7) final revision, 
and (8) validation. 

It is a way of learning and a way of pre- 
senting materials to be learned. It is essen- 
tially self-instructional, meaning that basic- 
ally the student is learning by himself and at 
his own pace with the help of a programed 
text or machine. Because it enables students 
to proceed independently, programed instruc- 



tion is the first teaching technique that per- 
mits breaking the traditional classroom lock- 
step procedure. 

Programed materials are based on a care- 
ful and detailed definition of learning objec- 
tives. To provide the learner maximum 
opportunity to attain these objectives, the 
subject-matter is broken into small, easily 
absorbed increments called “frames, ” vdiich 
are carefully selected and sequenced to build 
on preceding units. 

In each frame the learner must do some- 
thing (active response) to show that he has 
grasped the information presented. The 
student can immediately check his answer 
against the correct one (confirmation). Thus 
the student knows viiether he has grasped 
the information. If his response is correct, 
the program reinforces his learning throu^ 
repetition; if incorrect, he can study the 
information again or ask his teacher where 
he went wrong. Having to act maintains his 
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attention, and knowing that he has acted 
correctly inaintains his confidence. 

Russell Pease, consultant for Dupont, in 
a talk that he gives on the subject, explains 
the value of this approach, ‘ - The increments 
of knowledge are readily understood and 
easily digestible. They are prepared so that 
a student^ s reply is almost always correct. 
The fact that the student is almost always 
correct encourages him. He enjoys his work. 
Learning becomes easy, pleasant and self- 
sustaining. The simplicity of this method is 
deceptive and the results are amazing.” 
Educational psychologists have found that man 
learns better and faster when he is con- 
fronted with a minimum of errors. Most 
programed instruction is geared to this end. 

In a very real sense, this technique 
narrows the communications gap between 
the students and the technical expert who 
wrote the course. It is almost like a tutorial 
relationship. Differences in age, education, 
job experiences, and learning speed tend to 
be neutralized - the learners emerge with 
more uniform levels of understanding because 
each sets his own pace and checks his own 
progress. By permitting flexibility hereto- 
fore impossible, programed instruction 
brings us closer than ever before to the goal 
of individualizing instruction. 

Illustration 

The best way to become familiar with the 
technique of programed instruction is to 
experience it yourself, if you have not already 
had the opportunity. With this thought in 
mind, the next page illustrates one type of 
programed learning. 

Underlying Educational and Psychological 
Concepts 

Programed instruction provides the first 
real penetration of psychological theory into 
the educationsll process. Here, in one of the 
few instances where the product of learning 
theory has found direct and practical appli- 
cation in the classroom, educational tlieory 
and a method of instruction converge. 

Ernest R. Hilgard, professor of psy- 
chology and education at Stanford University, 
has described six established principles from 
the psychology of learning which support 



programed instruction. 

1. Programed instruction recognizes in- 
dividual differences by beginning where the 
learner is and by permitting him to proceed 
at his own pace. Individual differences in 
learning ability have long been observed in 
all areas of intellectual and motor activity. 

It is possible that programed learning may 
succeed in reducing large Individual differ- 
ences because of these features. 

2. Programed instruction requires active 
participation on the part of the learner. 
Learning By doing is an old educational 
slogan, and is still a good one. Active par- 
ticipation by the learner leads to faster and 
more effective learning than does the more 
conventional learning situation. 

3. Programed instruction provides for 
immediate knowledge of results . Whether it 
is because programed learning provides re- 
inforcement, reward, or cognitive feedback, 
there is abundant evidence that immediate 
knowledge of results is important in learn- 
ing. It favors learning the right thing while 
preventing the repetition and fixation of 
wrong answers. 

4. Programed instruction reduces anxiety 
that so often is connected with the learning 
situation. The learner is not threatened by 
the task because he knows he can learn and 
is learning. This knowredge brings him 
satisfaction and increased confidence in his 
ability. The major principle involved is that 
learning motivated by success is more effec- 
tive than learning motivated by failure or 
fear of failure. 

5. Programed instruction provides spac- 
ed review in order to guarantee the hi^ 
order of success that has become a standard 
requirement of good programs. If properly 
arranged, review permits a hi^ degree of 
learning on the first run through a program. 

6. Programed instruction emphasizes the 
organized nature of knowledge so often lack- 
ing with other methods of Instruction. Good 
programs are written to provide for good 
continuity between the easier (earlier) con- 
cepts and the harder (later) ones. The proc- 
ess of programing requires examining the 
subject matter very carefully in order to find 
out what has to be known before something 
else can be learned, and eliminating side 
issues that do not lead to cumulative learn- 
ing. 
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AN EXAMPLE OF LINEAR PROGRAMED INSTRUCTION 



This page will demonstrate how programed learning 
works. Cover up the answers in the outside columns 
as you work through the “frames" of information. 
Check answer to each frame before going on to the 
next. (Reprinted, with minor changes, by permission 
of Office Equipment and Methods Magazine.) 



1. Programed instruction learning of 
the linear type, as originated by Dr. 
B, F, Skinner of Harvard, breaks down 
subject matter into small chunks of 
information called frames. The unit of 
instruction in such a system is called 
a 






2. Each frame presents some new 
piece of information and calls for a 
response from the student. Therefore 
each frame calls for tivo actions. The 
student must read the 
and make his 


frame 




3. This participation, or learning by 
doing, is the most efficient form of 
learning. So the student is assisted by 
his own 


information 
or frame 

response 




4. One way of getting a response from 
the student is to ask him to complete 
the blank space in a sentence. By fill- 
ing in the space he is made 

to 


participation 




5. When a student has answered the 
question in a frame, he is given rein- 
forcement. After answering the ques- 
tion. his learning is by the 

correct answer. 


blank 

respond 


6. Periodically, a program will include 
a frame reviewing information that has 
gone before and calling for a series of 
responses to help fix the subject matter 
in the student’s mind. Just like this: 
The unit of instruction in a program is 
called a . It presents 

and calls ^br a 


reinforced 


from the student. This may be doneSy 
filling in the in a sentence. 

The student’s learning is then 
by the correct answer. 


7. The correct answer in a programed 
text can be conveniently placed along- 
side the next frame, as it is here. 
When the student has answered a ques- 
tion he turns to the to 

check his answer. 


frame 
information 
response 
blank space 
reinforced 



8. A well- written program will draw 
the correct answer from a student 
most of the time. The student should 
answer almost all the questions 


next frame 


9. A program should do more than 
merely present information. It must 
teach the student. A program fails if 
it does not and it 

must be revised. 


correctly 


10. Programed learning encourages 
each student to work at his own speed. 
He is not held back or rushed. He can 
mQyA olons st6s.dilv 


teach 


11. Programs can be written for pre- 
sentation in text form or for a teach- 
ing machine. A student can derive the 
same benefits from a as he 

can from a 


his own speed 




12. Either way he is given reinforce- 
ment after his response by being told 
the correct answer. His 
is obtained from the correct answer 
which he checks after making his 


text 

teaching 

machine 


13. Whichever method of presentation 
is adopted, it is the program not the 
presentation that is important. How- 
ever slick the presentation, it will fail 
if the is badly written. 


reinforcement 

response 


14. Programed learning is as good as 
the program and the program is as 
good as the programer. So the success 
or failure of a programed course 
denends on the skill of the 


program 


15. There are other forms and varia- 
tions - but this small program should 
give you an idea of some of the basic 
principles. 


programer 
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Distinguishing Characteristics 

At this point you may askj * ‘What features 
distinguish programed materials from text- 
books, workbooks, and other common teach- 
ing materials?** Persons well versed in 
programed instruction would describe it as a 
means of instruction with most, if not all, of 
the following characteristics: 

1. Precise definition of objectives in 
terms of learner behavior, the achievement 
of v^ich can be measured. 

2. Presentation of carefully organized and 

sequenced information . 

3. Arrangement”ol material in relatively 
small steps, each building on the preceding 
one, using steps that have been researched 
to provide almost errorless learning enroute. 

4. Individualized instruction - the oppor- 
tunity for each student to proceed at his own 
pace. 

5. Structuring so as to elicit frequent 
responses, making the student an active par- 
ticipant throu^out the learning experience. 

6. Provision for immediate feedback , al- 
lowing for confirmation or correction of 
response. 

7. Repeated testing or tiyout with the 
target population and consequent revision, to 
insure that learning outcomes have been 
achieved. 

8. By reason of the above, an almost 
errorless, hipdily successful path to the 
desired learning. 

These characteristics allow most users 
to specify a minimum acceptable standard 
for programed instruction of “90/90.** This 
means that 90 per cent of the students must 
pass the final test with a grade of 90 or 
better. If they do not, there is something 
wrong with the program - not with the stu- 
dents. 

These characteristics also allow a student 
of lower I.Q. usually to make as good a score 
as his classmates; it just takes him longer. 
This does not mean that “anybody can learn 
anything** but it is abig step in that direction. 

Research Evidence 

No method of instruction has ever come 
into use surrounded by so much research 
activity. Wilbur Schramm, director of the 



Institute for Communication Research at 
Stanford University, reviewed the research 
on programed instruction and summarized 
his findings in an annotated bibliography, 
“The Research on Programed Instruction.** 
Many of the research conclusions presented 
here are drawn from his publication. 

Do students learn from progr^ed in- 
struction? Schramm concludes “the re- 
search leaves us in no doubt of this.** They 
do, indeed, learn from programs on machines 
and from programs in texts. Every kind of 
student that programs have been tried on has 
been able to learn - pre-school, elementary, 
secondary, college, adult, professional, skill- 
ed, unskilled, military, deaf, retarded, and 
prisoners. Using programs, these students 
have been able to learn math, science, history, 
spelling, electronics, business skills, read- 
ing skills, basic industrial skills, and many 
other subjects. 

How well do students learn from pro- 
gramed instruction as compared to conven- 
tional kinds of instruction? It is difficult to 
generalize from comparative research be- 
cause it is impossible to define a “normal 
classroom teaching situation.** The outcome 
of such research often depends on what kind 
of teacher is being compared to what kind of 
program. 

Keeping these limitations in mind, we 
turn to reporting the results of the evaluative 
studies made. An analysis by Hartley (1966) 
indicates that existing programs are over- 
whelmingly as effective as, or more effective, 
than traditional forms of instruction. (Figure 

4 ) 

Such results are even more astounding 
when one realizes that a majority of the 
commercially published programs have not 
been adequately tested. Komoski (1966) re- 
ported on the testing experience of 291 self- 
instructional programs currently marketed in 
the United States. Records show that 36 per 
cent were given no field testing, 32 per cent 
were tested only once, and 32 per cent were 
tested more than once. 

What is the effect of immediate knowledge 
of results? A majority of the studies onpro- 
gramed learning support the theory that 
immediate knowledge of results contributes 
to learning. 

There are many areas in which the 
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Figure 4. Results of 112 Studies Comparing Programed and Conventional 
Instruction on General Effectiveness (Achievement and Efficiency) 



/ 




research is not conclusive. Logic and cau- 
tion are needed when making decisions on 
matters v4iere only limited research is 
available. Following is a brief summary of 
some of these problem areas. 




Some schools have constructed study carrels 
to help minimize distractions that may in- 
terfere with the study of programed mate- 
rials. Photo courtesy of Albert Salerno, 
Mahoning Valley Vocational School. 



What type of programing technique is best? 
By far the most common are the linear and 
branching types, each of which may employ 
numerous variations. These two and a less 
commonly known third type called **mathe- 
tics^^ are discussed in more detail later in 
this chapter. The great majority of studies 
on programed instruction has been done with 
linear programs. Where comparisons have 
been made, the evidence does not suggest 
any clear-cut superiority of the linear pro- 
graming method over the branching, or vice 
versa. Rather, both approaches appear to 
have certain strengths and weaknesses de- 
pending on the individual student and the 
tasks to be learned. 

Are machine presented programs more 
effective than programed texts ? Some pro- 
grams require a machine for their presen- 
tation, while some are available in two 
separate versions, one in book form and the 
other for use with a machine. In any case, 
it should be emphasized that the so-called 
teaching machines, in themselves, do not 
teach. The teaching is dependent on the 
programed materials which are presented 
through the machine. Research has not 
provided conclusive evidence of outright 
superiority of either machine or non-machine 
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presentation of printed materials. The choice 
of text or machine presentation probably will 
have to be made along pragmatic lines and 
in terms of the educator^ s specific purposes. 

Much research on programed instruction 
has dealt with questions concerning the char- 
acteristic variables of a linear program. 
What does the research say about these 
characteristics? 

(a.) Size of step . When significant differ- 
ences have been found in learning from pro- 
grams utilizing different step size » programs 
with short steps are generally favored. Some 
reports indicate, however, that superior 
students often become impatient and bored 
with long programs made up of short steps. 



(b.) Overt versus covert response . The 
overt response technique requires the stu- 
dent to recall and write out his answer, 
while the coveii: response technique requires 
recognition or selection of the correct re- 
sponse from a multiple choice of answers. 
Most studies of this variable report no sig- 
nificant differences. 

(c.) Individual pacing. Most of us would 
agree intuitively that students will learn 
more efficiently at their own pace. Never- 
theless, the experimenters have not yet been 
able to demonstrate as much advantage for 
individual pacing as might be expected. Some 
studies have indicated that self-pacing worked 
best for superior students. 




While adult trainees study their programed texts an Instructor assists Individual students 
whenever help is needed. 



Photo courtesy of Dupont Company. 
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Controversies 

Disagreements surround almost all new 
techniques. Programed instruction is no 
exception, and many controversies have 
arisen. Two of these issues merit further 
consideration here. 

(a.) Books versus machines . Basically, 
a teaching machine is simply a device used 
to convey information using ^e technique of 
programed instruction. A common view, 
especially a few years ago, was that pro- 
gramed instruction required the purchase of 
expensive machines. As Komoski says, “It 
is regrettable that audio visualists, out of an 
honest desire to positively exploit any and all 
devices that mi^it improve instruction, were 
the first group within education to actively 
promote programed instruction (or as they 
saw it, the teaching machine). The fact that 
this group, with its strong machine orienta- 
tion, was closely associated with the new 
technology served to create the impression 
that the technology was necessarily machine- 
based. This inevitably helped fortify the 
apprehension of the rest of the educational 
community that the incipient technology con- 
tained more of what was threatening than it 
did of what was hopeful and promising.^^ 

Several machine manufacturers today 
publish their programs in two forms, as 
sheets to be used in their machines and as 
programed textbooks foruse where a machine 
is not necessary or desirable. One supplier, 
Grolier, provides the following evaluation: 
“The primary utility of the machine is in 
the area of motivation and control. The 
machine is often effective in motivating the 
slow or reluctant learner. The control 
feature is most useful in unsupervised study 
situations and in educational research appli- 
cations.^^ 

Certain problems must be confronted 
immediately by anyone desiring to buy teach- 
ing machines. A great variety of machines 
is available, and there is little standardiza- 
tion among them. In evaluating any make or 
model of teaching machine, you must also 
assess the type and quality of progrems 
available for use in it. The mechanical 
dependability of many machines cannot be 
taken for granted. Potential purchasers 
should determine what maintenance problems 
have been encoimtered and the extent to 



which parts and service are locally available. 
Although programed textbooks may cost sev- 
eral dollars for a semester-length course, 
prices for teaching machines range from 
several dollars to several thousand dollars. 




An instructor assists one student in the load- 
ing of her teaching machine while other 
students ‘continue to study at their own pace. 

Photo courtesy of Welch Scientific Com- 
pany. 

Although many kinds of “machines^ ^ - 
computers, television, tape recorders, and 
film projectors, among others - are being 
used to present programed materials, it must 
be emphasized that machines used in the 
automation of instruction are only vehicles 
for presenting the instructional program to 
the learner. 

(b.) Types - Linear, Branching, and 
Mathetic^^ Throughout most of its short 
history, there has been general acknowledg- 
ment of two techniques of programing, the 
linear method and the branching or intrinsic 
method. Lately another technique, known as 
mathetics, has begun to gain acceptance as 
an efficient and tested method for imparting 
knowledge and skills. Althou^ there are 
proponents for all three techniques w^o will 
argue that their particular approach is best, 
the research thus far available fails to indi- 
cate superiority of any one technique for all 
situations. No two tasks present the same 
teaching problem and no single teaching 
technique can solve every problem. 
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The majority of current programs are 
linear. Characteristics of linear programing 
are; (a) frames are very short, usually not 
more than two or three lines, (b) student 
must use recall to construct the response, 
(c) basic premise is that the learner should 
be successful at all times, (d) student pro- 
ceeds throu^ each step from beginning to 
end. Some linear programs incorporate an 
** express-stopping^^ technique which directs 
a student to skip ahead if he has demonstrated 
advanced knowledge. 

The advocates of branch programing be- 
lieve that the student can learn efficiently 
from errors as well as from responding 
correctly, and that he can absorb a relatively 
large quantity of information at one time. 
Characteristics of the branching technique 
are: (a) frames are large, often include 
entire paragraphs, (b) student is asked to 
recognize the correct response usually from 
a multi- choice type question, (c) assumption 
made that student learns from successes and 
mistakes, (d) student is individually handled 
by use of sub- sequences which branch off 
from the main line; a failure at a crucial 
point leads to alternate remedial materials. 
In most branching programs the program is 
constructed so the choice of a particular 
answer to a diagnostic question determines 
which frame will be presented next. 

Mathetics is a relatively new programing 
technique being popularized by Teco Instruc- 
tion (Ft. Lauderdale, Florida). Its format, 
when frames are used, resembles the branch- 
ing technique. Individuation exercises are 
included, vhich may be skipped entirely by 
those already proficient but permit more 
instruction for those \dio need it. A major 
feature of this technique is the degree of 
task simulation used. Advocates hold that 
the greater the simulation, the greater the 
transfer. Therefore, many mathetical pro- 
grams employ kits or simulators \\iiich 
include components quite similar to those 
v/hich are used in the actual task. High costs 
of simulation have presented some economic 
problems. Printed matter in mathetics form 
is usually profuse with diagrams and pictures 
in varying stages of completion. According 
to its advocates, the active responses re- 
quired of the student during the program are 
always, to the degree required, simulated 
performance ox the task he is learning. 



For example, if the task were inspecting 
steel drills, the kit accompanying the pro- 
gram would include a drill gauge and some 
actual drills ground in various ways. In- 
corporated into the program at appropriate 
places would be instructions which tell the 
student to pick up his drill gauge and check 
specific drills for lip clearance, lip angle, 
or whatever, depending on the point being 
made by the program. 

There is little basis at present to favor 
any one of these general types over the 
other. Specific types of programs are likely 
to be useful for particular teaching problems. 
In some instances the most effective program 
may involve a combination of these different 
styles. 

Some Advantages 

Althou^ many of the advantages of pro- 
gramed materials have already been men- 
tioned, others have not been. For the con- 
venience of the reader, those considered 
worthy of special note are summarized here. 

1. They are self-paced, permit each 
learner to move throu^ the sequence of 
responses at his own speed, and thus ap- 
proach private tutoring. 

2. They can free the teacher from the 
routine and drilbnaster tasks of instruction 
and provide him with more time for creative 
and interpersonal activities with students. 

3. They can successfully teach most kinds 
of information and certain skills. 

4. They are efficient - unnecessary verb- 
iage is eliminated and only information cru- 
cial in terms of the program's stated objec- 
tives remains. 

5. They are based on sound theories of 
education and psychology. 

6. The information presented is organized 
and sequenced for individual readiness. 

7. Once oriented and motivated by the 
teacher, the student can learn alone with a 
good program. 

8. A single teacher can monitor and help 
individual students who are working on a 
variety of subjects at the same time. 

9. Learning tends to be of higher quality 
for all students because of the individual 
pacing and better control over what is pre- 
sented and how it is presented. 

10. The low error rate of most programed 
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materials is a great motivational tool in 
itself - especially for slower students who 
may be successful at learning for the first 
time. 

Several other benefits of programed in- 
struction have been realized by industrial 
giants like Dupont and the Proctor and 
Gamble Company. Most industrial training 
involves adults, who, for the most part, are 
seriously motivated by the desire to improve 
their rank and stature within the company. 
Whether these gains will be possible in 
occupational and other public educational 
situations still remains to be seen. With 
these precautions aired as a check against 
overenthusiasm the following industrial bene- 
fits are listed: 

(a.) A 25-50 per cent reduction in train- 
ing time. 

(b.) 10-25 per cent more effective learn- 
ing. 

(c.) Savings of $30 to $60 per student per 
course, as compared to conventional training 
methods. 

(d.) Uniform high quality instruction 
which is transmitted to job performance. 

Some Limitations 

Despite the fact that programed instruc- 
tion has shown great potential, it is like 
most any other technological improvement in 
that it also has some limitations and dis- 
advantages. 

1. It cannot take the place of good schools 
or skilled teachers. 

2. It cannot solve the problems of over- 
crowded classrooms. 

3. It cannot succeed in classrooms where 
teacher attitudes toward it are hostile. 

4. It cannot provide effective instruction 
imless the materials selected are properly 
prepared and tested. 

5. Some students become bored after 
working with programs for some length of 
time. 

6. Administrative problems of scheduling 
may arise when students using programed 
instruction and finishing at different times 
are scheduled for subsequent training as an 
intact group. 



7. Teachers who will use programed in- 
struction must be trained in the use and 
classroom management of such materials. 

8. Selection of good programs that will 
fit in or complement the existent curriculum 
is not easy. 

9. As with any technology, certain costs 
are involved - acquiring the program, train- 
ing teachers to use it, and evaluating it. 

10 . The number of good programs available 
commercially is quite limited in some con- 
tent areas . One survey indicated that approx- 
imately sixty per cent of the programs avail- 
able were in the areas of mathematics, 
science, and English. 

Student Reactions 

Student evaluation of and comments about 
a new medium of instruction provide a good 
indication of how well they accept it and, in 
turn, an indication of how well they are likely 
to learn and like learning from it. The eval- 
uation summary and student comments quoted 
below are reprinted through the courtesy of 
Mahoning Valley Vocational School, Vienna, 
Ohio, and its basic education supervisor, 
Albert A. Salerno. This school was created 
to provide saleable vocational skills for 
young men aged 16-21 who are either un- 
employed orxmderemployed. The Programed 
Learning Center at Mahoning was opened in 
August 1965. Using programed instruction in 
the Center atmosphere was another step 
toward ‘^customized education” writes Mr. 
Salerno. Their philosophy of customized 
education is “an attempt to design a com- 
plete educational program for each of the 
students at the student^s achievement level.” 

The student evaluation sheet used by a 
random sampling of 100 students at Mahoning 
at the end of one year is reproduced in Fig- 
ure 5. The figures represent a summary of 
the students^ responses. The evaluation sheet 
includes two items each on magazines and 
music M^iiich were included to sample student 
reaction to vocational magazines available in 
their library and to the playing of classical 
or semi-classical music three to four hours 
during the day. 
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Figure 5: STUDENT EVALUATION SHEET - PROGRAMED LEARNING CENTER 

Please check as many boxes concerning your thou^ts on the Program^ Learning Center 
as you wish. We are very interested in your reactions to Programed yarning ^ 
Center. You may also want to add some of your own ideas in the space allotted at 
of the page. Naturally it isn't necessary for you to sign your name. Mr. Albert Sal^ — 



**It helped me, and I 
learned a lot" 



77 ** Explained clearly" 



59 ‘‘Don't like the music" 



“Didn't help" 



0 “Would like P.L. in 
other classes." 



52 “Like the music" 
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“Liked it" 



89 “Made me think" 



61 “Am in P.L. too many 
hours" 



“Waste of time 



9f 



0 “Know immediately why 57 “Should be open at 
I'm right or wrong" nite" 
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“Interestin r" 
“Boring" 



84 “Liked working alone" 72 “No need to open at nite' 



4 “Helped in my Vocational 64 “Like the magazines 
class" — 



ff 
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“Prefer to classroom 



32 “Didn't really help in 
my Vocational class" 



9 “Don't like the maga- 
zines" 



“Good Review" 



60 “Good study atmos- 



69 “Am in P.IXl. too few 
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Efficient" 



58 “Poor place to do any 
studying" 



“Didn't like it" 0 “N ot enough teacher help" 6 

This evaluation sheet was filled out by a random sampling of 100 students. 

THIS SPACE IS FOR WRITING ANY ADDITIONAL COMMENTS THAT YOU K^YB^VE CON- 
CERNING THE CENTER: (Note: This is a summary of the actual comments written by tne 

100 students.) 

“This kind of program in high school would have helped the slow student gain confidence. 
“Best equipped study center I have ever been exposed to." 

“Quiet and convenient enough to do anything." 

“Very helpful to me in Algebra." 

“Teaches the student howto help himself." , „ 

“To the rest of the guys I've talked to P.L, has helped in their vocational areas. 

“Need more books." 

“Good chance to review." 

“Got a greater variety of magazines." 

“There is no one on my back, if I feel uninterested I can do something else. 

“Can ieam better at my own speed." 

“First time I've had a self-tutor program and it is challenging. ^ »» 

“All trainees should take advantage of this. It teaches the fimdamentals of a subject area. 
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CHAPTER III 



SELECTION AND USE OF PROGRAMED MATERIALS 



This chapter will cover only briefly 
certain topics related to the selection and 
use of programed materials. The brevity 
should not be regarded as an Indication of 
the importance of the points covered. Se- 
lecting the best program available for your 
teaching objectives is hi^ly important and 
not an easy task. And even the best pro- 
gram needs to be properly integrated into 
the existing curriculum and properly ad- 
ministered if its full potential is to be 
realized. 

What Can Programed Materials Teach? 

In theory, any subject that can be verbal- 
ized and/or illustrated in one way or another 
can be programed. Programing helps people 
learn simple skills such as accurate meas- 
urement with a micrometer, and complex 
skills such as calibration of a precision 
instrument. It is possible through a self- 
instructional program to teach a rather 
complex combination of cognitive and motor 
skills - the reading and interpretation of an 
electrocardiogram tracing, for example. 

Programed instruction has been most 
successful to date in teaching cognitive 
material - facts, principles, concepts. 
These programs can be used from elemen- 
tary to college levels for a great variety of 
subjects. One guide to available programs 
which classifies them by subject lists over 
80 different subject matter areas. 

Industry has made extensive use of pro- 
gramed materials in teachingbasic industrial 
skills. A study in which this writer was in- 
volved raised serious question as to whether 
programed materials alone can satisfactorily 
teach a complex psychomotor skill such as 
regrinding a drill. The students had no 
difficulty in learning the necessary cognitive 
material from the program but had con- 
siderable difficulty performing the motions 
required for proper regrinding. 

When teaching a skill, some programs 
call for the student to read each step in the 
book and then perform it with the actual 



equipment or simulated equipment. Similar- 
ly, the program may be supplemented with 
illustrations, photographs, samples, or other 
devices. These are usually called panels or 
exhibits and are designed as an integral part 
of the program. This procedure is basic to 
the mathetics programing technique and to 
this writer's knowledge is very effective. 
After such self-instruction, the learner us- 
ually performs the task in a real situation 
under the observation of a qualified instruc- 
tor. 




Visual techniques are sometunes used to pre- 
sent vocational subject matier. 

Photo courtesy of Albert Salerno, Mahoning 
Valley Vocational School. 



How Should Programs Be Used? 

Much remains to be learned about how a 
program can best be used in a course. One 
fact is certain - programs can be used in a 
variety of ways. They may be employed as 
the main source from which students are 
expected to learn facts, principles, or certain 
skills. Some feel there is a place for pro- 
grams which will even teach a subject with- 
out an instructor. These might be appropriate 
for adults or for remedial courses designed 
to provide review for students entering 
college without sufficient preparation. 
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other programs should be used to teach 
parts of courses, to introduce other instruc- 
tion, or to provide enrichment material for 
students with special interests and abilities. 
In most schools, programs should be used in 
conjunction with other media as one of the 
means of teaching a course. 

The NEA sound filmstrip, “Selection and 
Use of Programed Materials,'' summarizes 
the seven most common uses of programs as 
follows: 

1. Basic instruction. 

2. Incidental instruction. 

3. Correlated instruction. 

4. Remedial instruction. 

5. Enrichment. 

6. Substitute instruction. 

7. Research. 

What Kinds of Materials Are Available? 

There has recently been a dramatic in- 
crease in the number of commercially avail- 
able programed texts in a variety of subject 
matter areas. More will be said later, but 
mere availability is no guarantee of quality . 

The most comprehensive list of pro- 
gramed courses available in this country has 
been produced by Carl Hendershot of Delta 
College, Bay City, Michigan. His publication, 
“Programmed Learning: A Bibliography of 
Programs and Presentation Devices," in- 
cludes programs covering over 80 subject 
matter areas. The current revision (1967) 
lists the addresses of 150 program publish- 
ers and includes over 2,500 programs with 
brief descriptive information about each. 
Pertinent data given include the title, author, 
approximate length, intended student popula- 
tion, and price. No attempt has been made 
to evaluate the programs, but the compiler 
has pointed out prerequisite knowledge and 
other facts to help determine the program's 
possibilities for various instructional situa- 
tions. 

In Appendix A this writer has compiled 
lists of publishers of programs relevant to 
occupational education. The listing does not 
include prices, titles, or other descriptive 
information. Because new programed mate- 
rials are appearing rapidly, all available 
programs may not be listed here. To stay 
well informed, you must subscribe to one or 
more of the general listings and/or request 



that individual publishers keep you on their 
mailing list. 

What can be done if there are no high 
quality programs available in the desired 
subject area? The answer of course is to 
consider the possibility of writing your own. 
Remeniber, however, that preparing a pro- 
gramed instruction course is a demanding 
and exacting task - and usually a long one. 
The typical program author needs about six 
months to complete his research, writing, 
testing, revisions, editing, and production 
before the course is ready to be used. Short 
courses in how to program (offered by the 
University of Rochester and the University 
of Michigan) are hi^ly recommended to any- 
one phinning to do much programing. 

The Denver, Colorado, school system has 
discovered certain advantages to encourag- 
ing some local production even though it 
depends on commercial materials for most 
needs. Locally produced programs may be 
more tailored to the needs of the particular 
school. And more importantly, they conclude, 
the principles of programing will tend to 
spill over and improve all teaching and all 
teaching materials. 

What Are the Economics of Using Programed 
Materials? 

Figxire 6, which shows the trend in costs 
of programed texts, was prepared by the 
editors of Programed Instruction . While in 
1963 slightly more than half of all programs 
sold for school use were priced at $4.00 or 
less, during 1964-65, 73 per cent of the pro- 
grams produced were sold in this cost range. 
This trend occurred in spite of the fact that 
during the same period the average length of 
programs increased. Most publishers are 
attempting to make programed inaterials 
more economically competitive with other 
kinds of instructional materials. One way 
this can be accomplished is by producing 
reusable programs which require only the 
purchase or duplication, if permitted, of stu- 
dent answer sheets in succeeding years. 

Teaching machines vary in cost from 
Jeveral dollars to several thousand dollars. 
Their costs have remained substantially the 
same probably because the high cost of some 
hardware and its fragility in student hands 
have resulted in a low sales volume. The 
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cost of programs designed for machine use 
has declined somewhat. 

Figure 6. Programed Text Costs 




Local or in-house production of tested 
high quality materials can be quite costly. 
Dupont estimates that it costs them approx- 
imately $3,000 to $4,000 per course hour for 
materials developed by company personnel. 
Most school teachers and administrators 
should carefully scrutinize commercially 
available programs before attempting to 
write their own. 

The cost of industrial training by pro- 
gramed instruction, as reported by the Re- 
sources Development Corporation, is sur- 
prisingly low - less than $1.00 per hour of 
training for materials developed for industry- 
wide audiences. This, they report, compares 
very favorably with the cost of assembled 
training - instructor time, classroom facili- 



ties, training materials, and other overhead 
which starts at $11.00 per hour per man. 
Needless to say, public educators have not 
been able to realize similar economic bene- 
fits. 

W hat Selection Procedures are Recommend- 
ed? 

Program evaluation is difficult. The 
benefits of programed instruction can be 
realized only if users have adequate infor- 
mation with which to evaluate self-instruc- 
tional materials. Since many programs of 
varying quality (from poor to excellent) are 
available, careful selection is crucial. Some 
practical su^estions are presented here 
which will assist teachers and administrators 
charged with this responsibility. Because of 
space limitations, this discussion is neces- 
sarily brief; however, evaluation is of such 
importance that in Appendix C a classic 
statement on ''Criteria for Assessing Pro- 
gramed Instructional Materials^ ^ is repro- 
duced. This statement was prepared by a 
joint committee made up of representatives 
from several professional organizations con- 
cerned about providing direction to the pro- 
duction, dissemination, and utilization of 
programed materials. 

The following steps are recommended as 
a procedure for evaluating a program. 

1. Read the program carefully and thor- 
oughly. - ^ ^ 

2. Compare the compatibility of its con- 
tent with your instructional objectives. 

3. Examine the adequacy of programing. 
The frames should present a careful, logical 
progression toward mastery of the subject 
matter. Students should be required to re- 
spond to critical aspects of each item or to 
perform the operation that step was meant 
to teach. The format should be an attractive 
one, appropriate to the subject matter and 
to your students. The reading level must be 
suitable. 

4. Examine the field test and other vali- 
dation data available. They should provide 
information on the programme effectiveness 
and efficiency. 

5. Experiment with the program on a 
small group of students before adopting it 
on a wide scale. 
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What Administrative Changes Will Require 
Attention? 

Effective use of programed instruction in 
any school system will depend upon admin- 
istrative assistance, guidance, and support. 
Charged with the responsibility for guiding 
and unifying the efforts of the school system 
toward specific objectives, the administrator 
will want to consider the following questions 
relating to the use of programed instruction. 

1. What orientation or training should be 
provided teachers before they use programs? 
Consideration should be given to holding in- 
service workshops for the faculty to acquaint 
them with the medium. 

2. What orientation should be provided 
students and parents? Some publicity about 
the nature of programed instruction and 
reasons for introducing it is recommended. 

3. What effects will the use of programed 
materials have on curriculum and schedul- 
ing? Certain curriculum changes may be 
desirable because of the greater opportunity 
for individualized instruction. Individual 
pacing may also require more flexible sched- 
uling; students starting a program at the 
same time will not all finish at the same 
time, 

4. What will be the financial needs? 
Probably an overall financial plan should be 
established, one which looks ahead to con- 
tinued use of programed materials. Consider 
the costs of acquiring programs, training 
teachers to use them, and evaluating them. 
Financial assistance is available from the 
federal government under several education 
acts. 

5. What logistical problems - space, 
building design, etc. - will need attention? 
Some schools are establishing special pro- 
gramed learning libraries where students 
can come, check out a programed text, and 
study during their free time. Most use pro- 
grams in their regular classrooms. Others 
have built carrels for individual study. 

6. What changes may be necessary in 
reporting pupil achievement? In most cases, 
regular course credits probably will be 
granted upon passing course exams, but 
additional credit may be granted to students 
who complete extra programed course mate- 
rial. 



What Changes May Occur in the Teacher* s 
Role? 

Thou^ his role may change considerably 
as more and more technology finds acceptance 
in the classroom, the teacher always has 
been and will continue to be the central 
figure in any instructional program. If, 
throu^ programed instruction and other 
technology, we free the teacher of the burden 
of routine instruction in facts, computations, 
and the like, the teacher will then have more 
time to do the things that only he can do. 
Programed instruction and teachers have 
different purposes. 

The following questions reflect some of 
the changes likely to occur in the teacher 
role as a result of using programed instruc- 
tion. 

1. How will the self-pacing quality of 
programed instruction affect classroom pro- 
cedures? Teachers will have to learn howto 
manage programs effectively, lest they be- 
come a burden instead of an aid. If a class 
of students are to finish a program at about 
the same time, then there will have to be 
considerable difference in the starting times. 
Use of programed materials must be planned. 

2. How will use of programed instruction 
affect the teacher^ s work in the classroom? 
Although programed materials will relieve 
the teacher of much that is routine, the 
teacher must still supervise use of the mate- 
rials. More time will be spent working with 
individuals or small groups when using 
programed instruction than was possible with 
conventional instruction. 

3. How can the teacher reinforce the 
program? By being thoroughly familiar with 
its content, by using good management pro- 
cedures, and by displaying a positive attitude 
toward its use, a teacher can contribute 
greatly to successful use of a program. As 
with any tsrpe of instruction, the teacher 
should develop an enthusiastic climate for 
learning. 

4. How will the teacher evaluate student 
progress? The method of grading and test- 
ing should be carefully explained to students. 
They must understand they are not being 
tested while working on the program itself. 
Regular testing and reporting procedures 
are easily adapted for use with programed 
instruction. 
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5. What should be the teacher»s responsi- 
bility with regard to programed instruction? 
Certainly the teacher should strive to become 
well informed about programed instruction 
and its uses. In most schools, teachers using 
programed materials will have a voice in 



their selection . Thus , the teacher must learn 
to distinguish between a poor program and a 
good one. He should want to attend in-service 
workshops to learn about program develop- 
ment, to get experience in writing frames, 
and to discuss their use in the classroom. 



CHAPTER IV 

SOME SUCCESSFUL APPLICATIONS OF PROGRAMED INSTRUCTION 



This chapter reports on several case 
studies where programed instruction has been 
successfully used. Case histories involving 
as many occupational education areas as 
possible have been included, although cases 
are not specifically identified as to vocational 
field. The histories include situations involv- 
ing elementary and secondary education, in- 
dustrial training, general and vocational 
subjects. 

1. Subject Area: Basic Industrial Skills 
Program(s): Varied 

Results: Dupont’s experience withprogram- 
ed instruction courses in 106 skills since 
1959 shows that this new technique trains 
operators 25-50 per cent faster and 10-25 
per cent more effectively. In the new tech- 
nique of programed instruction, they found a 
training tool that costs $30-$60 less per 
student per course compared to conventional 
training methods. 

Source: Dr. Arthur Santora, Head, In(festrial 
Training Service, Dupont Company, Wilming- 
ton, Delaware. 

2. Subject Area: Business 

Program (s): Several texts using themathet- 
ics design (including simulated practice ex- 
ercises) to train bank tellers. 

Results: Retention immediately improved 
and the amount of verbal instruction was 
reduced from 48 hours to 6 hours; resulting 
in a shortening of the average training per- 
iod from five weeks to as little as three 
weeks. 

Source: Gordon Rhodes, Director of Train- 



ing, First National City Bank, New York, 
New York. 

3. Subject Area: Elementary 
Program (s): Varied 

Results: “Experimentation conducted thus 
far supports the expectation that good pro- 
grams, carefully developed, can significantly 
improve the quality and economy of instruc- 
tion. Whether any particular program will 
do so is subject to question until established 
by adequate tests of that program.” 

Source: Programed Instruction Project - 
Annual Report, 1965-66, Board of Education 
of the City of New York, New York. 

4. Subject Area: English 

Program (s): Grammar from commercial 
source, one year length. 

Results: Respondent reported, “Program 
used with accelerated classes. This method 
of teaching English grammar was found to be 
as good or better than the conventional 
method for these classes. The method con- 
sumed less Instructional time.” 

Source: Warren, Ohio Public School system 
as quoted in NEA’s Educational Research 
Service Circular #7, 1966. 

5. Subject Area: General 
Program(s): Varied 

Results: Dr, Carl Hendershot, consultant to 
business, industry, colleges, and schools, 
has experienced extensive association with 
the selection and use of programed instruc- 
tion. At Delta College, programs have been 



19 



r> 



o 



used under his direction since 1961 as basic 
texts, supplementary texts, and for both adult 
and student self-instruction. In all of these 
applications, programed texts selected to 
meet the requirements of instruction and the 
needs of the student have produced highly 
desirable results. 

Source: Coordinator of Improvement, Delta 
College, University Center, Michigan. 

6. Subject Area: Health- Medical Education 

Program (s): A linear program on * ‘Allergy 
and Hypersensitivity” 

Results: Announced in 1963, the demand for 
this publication by individuals, medical 
schools, local, medical societies, specialty 
groups, and others has been so extensive 
that over 200,000 copies have been dis- 
tributed. More than two-thirds of the med- 
ical colleges are using the program as an 
instructional segment of their curricula. 
Source: Dr. Jerome Lysaught, College of 
Education, University of Rochester, Roch- 
ester, New York. 

7. Subject Area: Health and Safety 

Program(s): A two-hour program on how to 
“positively reinforce” employees for proper 
lifting. 

Results: Program was administered to su- 
permarket supervisors in two widely sep- 
arated parts of the country. Before the 
training, 39 back injuries were reported in 
a six-month period; only three were re- 
ported in the six months following training. 
(Average cost of each back injury estimated 
at $4,050.) 

Source: Brochure prepared by Center for 
Programmed Learning for Business, The 
University of Michigan, Ann Arbor, Michigan. 

8. Subject Area: Home Economics 

Program(s): Nutrition, from commercial 
source, one semester length. 

Results: Respondent reported, “We received 
reasonably good results from this program 
in meeting a specific need. Programs cen- 
tering on a specific topic seem more suc- 
cessful than total programs.” 

Source: Gary, Indiana, Public School system, 
as quoted in NEA's Educational Research 



Service Circular #7, 1966. 

9. Subject Area: Industrial Training 
Program(s): Varied 

Results: Dr. Garland S. Wollard, director 
of education for the Federal Bureau of Pris- 
ons, reports that the experiences gained at 
the Petersburg Reformatory have been so 
encouraging that he plans to introduce pro- 
gramed instruction to the other five federal 
reformatories as well as to two federal 
penitentiaries. One of the by-products of the 
Petersburg experiment was that inmates 
felt they could do something to improve 
themselves. This replaced a feeling that 
they were being “used” by the institution. 
Source: Article in New York Times by Gerd 
Wilcke, spring 1967. 

10. Subject Area: Industrial Training 
Program(s): Varied 

Results: The Boeing Company first offered 
programed instruction on a voluntary trial 
basis in January 1966, Results have been 
such that they expect to enroll 10,000 stu- 
dents in 50 courses in 1967. Employees have 
to pay for their own text, study the course 
off-hours at the time and place of their own 
choosing, and at their own pace. Some over- 
all conclusions have already been made: (a) 
A large majority of their employees likes the 
programed instruction method. They want to 
learn whenever and wherever they choose, at 
their own speed, away from the classroom, 
(b) Future course enrollment and interest in 
programed instruction and adult education 
will continue to accelerate. 

Source: Article in “Training in Business 
and Industry” by Paul Watson, Programed 
Instruction Coordinator, The Boeing Com- 
pany, Seattle, Washington. 

11. Subject Area: Medical-Nursing-Sales 

Program(s): Two programs, each approxi- 
mately six hours long, one on a complex 
disease and the other on a drug to combat it. 
Results: New company salesmen were re- 
luctant to discuss this drug with physicians 
xmtil they had been on the job six to nine 
months. After completing the program, new 
salesmen now approach the physician on the 
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particular drug within the first month they 
are on the job, resulting in new sales per- 
sonnel being productive six months sooner. 
Source: Brochure prepared by Center for 
Programmed Learning for Business, The 
University of Michigan, Ann Arbor, Michigan. 

12. Subject Area: Personnel Practices 

Program(s): One and a half hour program 
developed to train supervisors in the use of 
a new complex computer form for personnel 
records. 

Results: Program was distributed to 5,000 
automobile manufacturing employees across 
the country before switching to new system. 
New system is functioning with no problems, 
even though errors had to be non-existent in 
order to insure its success. 

Source: Brochure prepared by Center for 
Programmed Learning for Business, The 
University of Michigan, Ann Arbor, Michigan. 

13. Subject Area: Trade and Industrial 

Program(s): Machine Shop - programed 
materials include slides, tapes, and printed 
materials - all teacher produced. 

Results: Respondent reported, “Has truly 
individualized instruction and made instruc~ 
tor available for greater personal super- 
vision. Also makes last hour class as effec- 
tive as first. 

Source: Independence, Missouri, Public 
School system, as quoted in NEA^s Educa- 
tional Research Service Circular #7, 1966. 

14. Subject Area: Selling and Retailing 

Program(s): “Making Courtesy Work for 
You^' and “Making Suggestions to Increase 
Sales. 

Results: The actual sales increases record- 
ed have convinced Allied Stores of the pro- 
grams^ effectiveness. Tests showed an in- 
crease in sales dollars per hour over the 
control group of 25 per cent by the pro- 
gramed group and 8.3 per cent by the class- 
room group. As a result of these tests, 
store personnel and training directors have 
been advised to use the programed manuals 
as the basic tools of retail sales training. 
Source: Article in “Department Store Econ- 
omist,’^ written by Frederick Finigan, Per- 



sonnel Director of Allied Stores, April 1965. 

15. Subject Area: Selling and Retailing 

Program (s): Thirteen separate self-study 
units on retail food store operations. 

Results: A research study was conducted by 
Western Michigan University to determine 
the effectiveness of the program. Tests 
were administered to a typical group of store 
employees before and after they studied the 
units. A substantial and significant improve- 
ment was found in the knowledge level of the 
group. The employees found the materials 
interesting, they found the learning easy, and 
they felt they had significantly increased 
their understanding of store operations. 
Source: Announcement brochure from Sales 
Training Department, The Quaker Oats Com- 
pany, Merchandise Mart Plaza, Chicago, 
Illinois 60654. 

16. Subject Area: Spelling 

Program(s): TMI-Grolier Spelling 
Resultfs: A controlled experiment was con- 
ducted using two classes of 30 students each 
selected from a cross-section of ability 
levels. One sixth grade group was given the 
program and the other was taught conven- 
tionally with textbooks. Students v^o used 
the program scored higher gains after eight 
weeks of study than the conventionally-taught 
class after twelve weeks of study. Pro- 
gramed learning saved time for above average 
students and eliminated the problem of loss 
of instruction resulting from illness as well 
as the problem of how to work with a new 
student in the class. 

Source: Research Department, Educational 
Divisions, Grolier Inc., New York. 

17. Subject Area: Vocational Agriculture 

Program (s): Parliamentary Procedure, five 
to seven hours. 

Results: A research study conducted at Cor- 
nell University studied the relative effective- 
ness of supplementing programed instruction 
with blocked versus spaced review. The 
program was administered to 279 first year 
vocational agriculture students in New York 
State. Conclusions drawn were that neither 
spaced nor blocked review resulted in sig- 
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nificantly better performance than was ob- 
tained from the program alone. 

Source: Final Report, * ‘The Relative Effect- 
iveness of Supplementing Programed Instruc- 
tion with Blocked Versus Spaced Review, 
by James A. Scanlon, Cornell University, 
May 1967. 

18. Subject Area: Vocational 

Program (s): Varied - 14 vocational areas 
Results: Programed courses at Mahoning 
Valley Vocational School offered students a 
chance to pursue vocationally oriented sub- 



jects at their own pace, with a minimum of 
trouble or error. The low error rates and 
questions- response- check format of pro- 
gramed instruction was a great motivational 
tool in itself. This type of learning situation 
actually built confidence and gave direction 
to students. Changes in skill behavior could 
definitely be observed in their vocational 
classes. An increase in the knowledge and 
skills experienced by our students was most 
evident on our post- test results. 

Three conclusions drawn by the coordin- 
ator of the Mahoning Programed Learning 
Center at the end of the first year were as 




Some vocational schools and many companies are establishing programed learning centers or 
libraries. Students may be allowed to select materials according to their interests or be 
assigned specific materials. Photo courtesy of Dupont Company. 
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follows: 

1. That programed texts were successful 
in vocational training. 

2. That students who are properly ori- 
ented and counseled can work for extended 
periods of time on programed courses. 

3. That both basic education and voca- 
tional skills can be developed or improved 
upon through programed instruction. 

According to the basic education super- 
visor, experimentation in this new educa- 
tional device has brought many benefits to 
Mahoning Valley and the vocational move- 
ment in Ohio. Vocational trainees are now 
better prepared for Jobs as a result of this 
success, Joint vocational schools in Ohio are 
now incorporating programed learning cen- 
ters in building plans, and Manpower De- 
velopment and Training Act city programs 
are incorporating programed learning cen- 
ters in Ohio. 

Source: Albert Salerno, Mahoning Valley 
Vocational School, P. O, Box 278, Vienna, 
Ohio 44473. 



It is impossible to present a complete 
picture of current developments and applica- 
tions of programed instruction. The inter- 
ested reader is referred to the following two 
reports which contain additional case studies. 



^Programed Instruction in Large School 
Systems^ ^ 

Educational Research Service Circular No. 
7, September, 1966. 

National Education Association, Washington, 

D.C. 

(Available for $1.75 each from Educational 
Research Service, 1201 Sixteenth Street, 
Washington, D. C.) 



Four Case Studies of Programed Instruction 
Fund for the Advancement of Education 
477 Madison Avenue, New York, New York 
10022 

(Dated Jime 1964, available free.) 



CHAPTER V 

RECOMMENDATIONS AND CONCLUSIONS 



Recommendations 

Programed instruction can be a success 
or a failure in any school system. Although 
the potential is great, programing is not a 
cure-all. Teachers and administrators must 
be careful in the use of programed instruc- 
tional materials. The following recommen- 
dations can help in effectively developing the 
potentialities of programed instruction. 

1. Learn as much as possible about pro- 
gramed instruction before you attempt to use 
it. See Appendix B for suggested readings. 

2. Seek the opinions of persons using 
programed instruction in your area, prefer- 
ably those teaching your subject. If this is 
not possible, write to teachers in other 
locations. 

3. Involve qualified teachers who are 
willing to learn about and try something new. 

4. Inform faculty members, students. 



parents, and others in the community about 
programed instruction and seek their help in 
evaluating it. 

5. Select materials very carefully. Good 
programs are challenging, not boring or 
insulting. The potential user of programed 
instruction should learn to discriminate be- 
tween an actual program and conventional 
instruction presented in a programed format. 
Remember, evidence based on student per- 
formance is needed to demonstrate that a 
program actually teaches. 

6. Try programed instruction on a small 
scale. Consider the project an experiment 
until it is proved successful in your school. 
Wide scale adoption can follow. 

7. Follow good management procedures. 
The programed instruction effort must be 
effectively managed. Without direction or 
purpose, its effects are unpredictable - 
sometimes it will work well, at other times 



o 



23 






the effect may be unclear, and sometimes it 
will fail miserably. 

8. Integrate programed instruction with 
other methods of instruction. Programed 
materials should be viewed as part of a vast 
array of instructional materials available to 
teachers and students. The full potential of 
programed instruction cannot be realized by 
its use alone; instead, the teacher must use 
it and other methods to supplement his efforts 
in the best possible way. 

C onclusions 

Programed instruction shows outstanding 
promise of making a definite contribution to 
education. An imcritical acceptance of pro- 
gramed materials or a premature or unin- 
formed rejection of them would be most 
unfortunate. All indications are that with 
imagination, resourcefulness, and adequate 
preparation in the use of this medium, all 
concerned will benefit. As teachers gain 
experience with the technique, the benefits 
and leamingprinciples involved in programed 
instruction also may be incorporated into 
other media of instruction. 

Programed instruction has been called 
* ‘infinite consideration for the learner. It 
represents the first real penetration of psy- 
chological theory into the educational proc- 
ess. Research conducted thus far supports 
the contention that good programs, carefully 
developed and properly administered, can 
significantly improve the quality and economy 
of instruction. 

The advantages of programed instruction 
outweigh its disadvantages. Its limitations 
must be recognized; it is not a panacea for 
all of education’s ills, nor will it revolu- 
tionize education overnight. However, its 
main advantages - individualized rate of in- 
struction, shortened learning time, more 
effective learning, and ability to relieve 
teacher fatigue - cannot be denied or ig- 
nored by educators. 

Programed materials offer a great 
amount of versatility in occupational educa- 
tion. They may be used as developmental, 
remedial, or accelerated tools depending 
upon the students and their needs. Adults 
as well can profit a great deal from self- 
instructional materials. The logical struc- 
ture and built-in success factor of programed 
materials offer students a chance to increase 
knowledge and skills at the same time mo- 



tivation to learn is being enhanced. 

The successes already reported by its 
users in both industry and education are 
widespread. It has gained acceptance in 
over 9,000 companies, an increasing number 
of general, teclmical, and vocational schools, 
the armed forces, and several government 
agencies, as well as in several foreign 
countries. 

At present programed texts offer the 
most economical route available to educators 
who want to move toward individualized in- 
struction. Although the costs of producing 
programs locally may be prohibitive for 
most school systems, the purchase of com- 
mercially prepared materials is not. With 
the latter, the problem is more likely to be 
finding a program of high quality in the 
specific subject matter area desired. 

A very skillful teacher can do v^atever a 
program can do and at least as well as the 
program can do it - with one exception. The 
teacher can tutor only one student at a time. 
Programed instruction is one of a variety of 
instructional devices that should be put at 
the command of the teacher to help him do 
his job better. The teacher has been and 
will continue to be responsible for decisions 
about what is taught and how it shall be 
taught. Neither the teacher alone nor the 
program alone can do the effective job that 
the proper combination of both can do. 

Following are a few predictions of what 
the future may hold for programed instruc- 
tion. 

1. History will someday record that pro- 
gramed instruction was the most effective 
development in giving access to learning 
since the invention of printing. 

2. 1975 will find programming a major 
technique of instruction at all educational 
levels, in all parts of the world. 

3. Although some school^ and teachers 
will prepare their own programs, in most 
educational systems teachers will use com- 
mercially prepared programs. 

4. Lastly, this writer agrees completely 
with the superintendent wiio said, “Although 
it is becoming evident that programed in- 
struction can teach without a teacher’s as- 
sistance, it seems to us that its greatest 
promise lies in its uses by an able, enthusi- 
astic teacher as one of several tools.’’ 

See for yourself - try programed in- 
struction in your school. 
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APPENDIX A 



PROGRAMED TEXT MATERIALS AVAILABLE IN OCCUPATIONAL EDUCATION 
SUBJECT MATTER AREAS AND THEIR PUBLISHERS 



Since programed text publishers often produce and sell materials in several subject matter 
areas, this listing has been divided into two parts to avoid unnecessary duplication of sources. 
Part I lists the subject matter areas and the code numbers given to publishers viio have 
materials available in that general area. These code numbers in turn correspond to a master 
list of 69 program publishers contained in Part II. 

No claim is made that all available subject matter areas and their publishers are listed. 
Neither does listing here imply that the author has evaluated the program. Rather, listings 
were made on the basis of general announcements made available to the author by publishers. 



PART I 

Subject Areas and Sources 



Adult Basic Education 
Source (s); 24, 41 

Agriculture - General 

Source(s); 18, 19, 58, 67 

Agricultural Mechanics 

Source(s): 9, 18, 21, 32, 53, 67 

Banking and Finance 

Source(s); 1, 2, 17, 22, 39, 64 

Blueprint Reading 
Source(s): 41, 69 

Bookkeeping and Accounting 

Source(s): 1, 3, 19, 21, 38, 41, 49, 66 

Business and Office Education - General 
Source(s); 3, 8, 28, 56, 59, 63 

Business English 

Source(s): 8, 9, 17, 19, 41 

Business Law 

Source(s): 1, 17 

Business-Machine Operation 
Source(s): 13, 22, 37, 46 
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Business Management 

Source(s): 8, 9, 17, 21, 22, 27, 38, 41, 50, 63, 68, 69 

Business Mathematics 
Source(s); 41, 59, 63 

Business-Office Practice 
Source(s); 8, 38 

Business- Typing and Shorthand 
Source(s); 3, 29, 38, 41 



Construction 

Source(s): 64, 69 

Data Processing 

Source(s); 5, 9, 17, 19, 31, 37, 41, 46, 48, 49, 68, 69 

Drawing and Graphics 
Source(s): 3, 18, 41 

Economics 

Source(s); 9, 10, 20, 21, 22, 41, 43 
Electricity 

Source(s); 1, 3, 17, 18, 23, 26, 29, 31, 38, 41, 49, 53, 68, 69 
Electronics 

Source(s): 3^_^15, 17, 20, 31, 38, 41, 49, 51, 65, 69 

Health j / 

Source(s): 10,^Jrtl, 21 

Home Economics 
Source(s); 21, 50 

Industrial, Trade and Technical Education 

Source(s); 3, 5, 6, 8, 9, 18, 23, 27, 38, 41, 53, 69 

Insurance 

Source(s): 1, 17, 52 

Mathematics- Arithmetic 

Source(s); 1, 14, 17, 18, 21, 28, 29, 30, 34, 39, 41, 42, 53, 56, 64 
Mathematics-Modem 

Source(s): 7, 10, 21, 29, 31, 33, 34, 39, 41, 43, 56, 68 

Mathematics-Slide Rale 
Source(s): 25, 30, 41 









Mathematics-Other Areas 
Sources (s): 1, 7, 10, 14, 

68 



17, 18, 20, 21, 25, 28, 29, 30, 33, 34, 41, 42, 48, 49, 51, 53, 57, 
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Measurement 

Source(s): 18, 38, 44, 53, 69 

Medical and Nursing 

Source(s): 7, 9, 16, 20, 21, 29, 36, 40, 42, 45, 47, 55, 60, 61, 62, 63, 68 

Parliamentary Procedure 
Source(s): 17, 30, 42, 57 

Personnel Practices and Supervision 
Source(s): 1, 5, 8, 19, 35, 38, 41, 53 

Safety 

Source(s): 10, 18, 53 

Science-Biology 

Source(s): 3, 7, 12, 14, 17, 19, 20, 21, 28, 29. 41, 42, 48, 55, 61 

Science-Chemistry 

Source(s): 1, 3, 7, 11, 12, 14, 17, 19, 23, 28, 29, 30, 33, 45, 49, 55, 61, 68 

Science-Other 

Source(s): 3, 7, 10, 14, 21, 23, 25, 28, 29, 31, 33, 34, 39, 40, 41, 43, 48, 53, 56, 68 

Selling and Retailing 

Source(s): 8, 9, 22, 37, 39, 46, 50, 63 



PART II 

Directory of Programed Text Publishers - 
: ydbject Matter Relevant to Occupational Education 

1. Addison- Wesiey Publishing Company, Inc. 

Reading, Massachusetts 01867 

2. The American Bankers Association 

90 Park Avenue, New York, New York 10016 

3. American Book Company 

55 Fifth Avenue, New York, New York 10003 
300 Pike Street, Cincinnati, Ohio 45202 

4. American Journal of Nursing, P. I. Reprints 

10 Columbus Circle, New York, New York 10019 

5. American Management Association, Inc. 

135 West 50 Street, New York, New York 10020 

6. American Society of Tool and Manufacturing En^neers 
20501 Ford Road, Dearborn, Michigan 48128 
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Appleton-Century-C rofts 

440 Park Avenue South, New York, New York 10016 

8. Argyle Publishing Corp. 

200 Madison Avenue, New York, New York 10016 

9. Basic Systems Inc. 

880 Third Avenue, New York, New York 10022 

10. Behavioral Research Laboratories 

Ladera Professional Center, Box 577, Palo Alto, California 

11. W. A. Benjamin 

One Park Avenue, New York, New York 10016 

12. Burgess Publishing Company 

426 South Sixth Street, Minneapolis, Minnesota 55415 

13. Burroughs Corporation 
Detroit 32, Michigan 

14. Central Scientific Company 

1700 Irving Park Road, Chicago, Illinois 60613 

15. Cleveland Institute of Electronics 

1776 East 17th Street, Cleveland, Ohio 44114 

16. Columbia University, Teachers College, Bureau of Publishers 
525 West 120th Street, New York, New York 

17. Doubleday & Company, Inc. 

501 Franklin Avenue, Garden City, New York 10017 

18. Du Pont Industrial Training Service 

Room 7450, Nemours Building, Wilmington, Delaware 19898 

19. Educational Methods, Inc. 

20 East Huron Street, Chicago, Illinois 60611 

20. Educational Systems Development 

31270 Stephenson Highway, P. O. Box 457 , Royal Oak, Michigan 

21. Encyclopaedia Brittanica Press 
425 North Michigan Avenue, Chicago, Illinois 60611 

22. Entelek hicorporated 

42 Pleasant Street, Newburyport, Massachusetts 01950 

23. Fearon Publishers, Inc. 

2163 Park Boulevard, Palo Alto, California 94306 

24. Follett Publishing Company 
1010 West Washington Boulevard, Chicago, Illinois 60607 
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25. W. H. Freeman and Company 

660 Market Street, San Francisco, California 94104 

26. Friden Inc. Service Training Center 

421 University Avenue, Rochester, New York 14607 

27. General Programmed Teaching, Division of Commerce Clearing House, Inc. 

424 University Avenue, P. O. Box 402, Palo Alto, California 

28. Graflex Inc. 

3750 Monroe Avenue, Rochester, New York 14603 

29. Grolier Educational Corporation 

575 Lexington Avenue, New York, New York 10022 

30. Harper & Row, Publishers Incorporated 

49 East 33rd Street, New York, New York 10016 

31. Hayden Book Co., Inc. 

116 West Fourteenth Street, New York, New York 10011 

32. Hobart Welding School 
Troy, Ohio 

33. Holt, Rinehart & Winston, Inc. 

383 Madison Avenue, New York, New York 10017 

34. Honor Products Company 

22 Moulton Street, Cambridge, Massachusetts 02138 

35. Human Development Institute 

1299 West Peachtree Street, N.E., Atlanta, Georgia 30309 

36. Instructive Communications Unit 

Training Branch, Communicable Disease Branch, Public Health Service, D.H.E.W., 
Atlanta, Georgia 30333 

37. International Business Machines Corporation 

6 Roosevelt Avenue, Endicott, New York 13760 

38. International Educational Services, Inc., Division of International Textbook Company 
Department 852A, Scranton, Pennsylvania 18515 

39. Learning Incorporated 

131 East Sixth Avenue, Scottsdale, Arizona 85215 

40. J. B. Lippincott Company 

East Washington Square, Philadelphia, Pennsylvania 19105 

41. McGraw-Hill Book Company 

330 West 42nd Street, New York, New York 10036 

42. McMahon Electronic Engineering, Research and Development Laboratory 
381 West Seventh Street, San Pedro, California 90731 
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43. The Macmillan Company 

866 Third Avenue, New York, New York 10022 

44. Model Publishing and School Supply Company 
1604 Hodiamont Avenue, St. Louis 12, Missouri 

45. C. V, Mosby Company 

3207 Washington Boulevard, St. Louis, Missouri 63103 

46. National Cash Register 

Marketing Services, Dayton, Ohio 45409 

47. National League for Nursing 

10 Columbus Circle, New York, New York 10019 

48. Pergamon Press Inc. 

44-01 - 21st Street, Long Island City, New York 11101 

49. Prentice-Hall Inc. 

Englewood Cliffs, New Jersey 07632 

50. Quaker Self-Study Program 

345 Merchandise Mart, Chicago, Illinois 60654 

51. RCA Institutes Inc. 

350 West Fourth Street, New York, New York 10014 

52. Research & Review Service of America, Inc. 

123 West North Street, Indianapolis, Indiana 46209 

53. Resources Development Corporation 
Box 591 , East Lansing, Michigan 48823 

54. University of Rochester, Rochester Clearin^ouse for Information on Self-Instruction in 

Medical Education 

University of Rochester, Rochester, New York 

55. W. B. Saunders Company 

West W’ashingion Square, Philadelphia, Pennsylvania 19105 

56. Science Research Associates, Inc. 

259 East Erie Street, Chicago, Illinois 60611 

57. Scott, Foresman and Company 

1900 East Lake Avenue, Glenview, Illinois 60025 

58. Scott Visual Aids Service 

110 Brantford Lane, Greenville, South Carolina 29605 

59. South-Western Publishing Company 

5101 Madison Road, Cincinnati, Ohio 45227 

60. Springer Publishing Company, Inc. 

200 Park Avenue, South, New York, New York 10014 
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61. Stipes Publishing Company 

10 Chester Street, Champaign, Illinois 

62. Superintendent of Documents 

U. S. Government Printing Office, Washington, D, C, 

63. Teaching Systems Corporation 

334 Boylston Street, Boston, Massachusetts 02116 

64. TECO Instruction Inc. 

3236 N. E. 12th Avenue, Ft. Lauderdale, Florida 

65. Tektronix, Inc. 

P. O. Box 500, Beaverton, Oregon 97005 

66. Tutorinc 

P. O. Box 432, La Porte, Colorado 80535 

67. Vocational Agriculture Service 

434 Mumford Hall, Urbana, Illinois 61801 

68. John Wiley & Sons, Incorporated 

605 Third Avenue, New York, New York 10016 

69. Xerox Education Division 

600 Madison Avenue, New York, New York 10022 
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APPENDIX B 



SELECTED BIBLIOGRAPHY OF RECOMMENDED REFERENCES AND 
LIST OF SOURCES FOR FURTHER INFORMATION 



Both the bibliography and the list of sources are highly selective. The lists contain val- 
uable reference materials and resource information. Several tjrpes of references are included 
and many of these contain extensive listings of additional programed instruction materials. 



Selected Bibliography 



A. Books 

1. Forbes, Jack E. A New Look at Programed Texts and Their Use . Encyclopaedia 

Britannica Press, 1965. 19 pp. 

2. Glaser, Robert (ed.). Teaching Machines and Programed Learning II ; Data and 

Directions. Department of Audiovisual Instruction, National Education Association, 
Washington, D. C., 1965. 831 pp. 

3 Hendershot, Carl H. Programmed Learning; A Bibliography of Progra ms and Pre- 
sentation Devices; 4th ed. Hendershot, 4114 Ridgewood Drive, Bay City, Michigan 
48707, 1967. 

4. Jacobs, Paul I., Milton H. Maier, and Lawrence M. Stolvrow. Guide to Evaluating 

Self-Instructional Programs . Holt, Rinehart and Winston, Inc., New York, 1965. 

84 pp. 

5. Lysaught, Jerome P., and Clarence Williams. Guide to Programed Instruction . Wiley, 

New York, 1963. 180 pp. 

6. Mager, Robert F. Preparing Objectives for Programed Instruction . Fearon, San 

Francisco, California, 1962. 62 pp. 

7. Programed Instruction in Large School Systems. Educational Research Service Cir- 
* cular, No. 7, National Education Association, Washington, D. C., 1966. 60 pp. 

8. Rummler, Geary A. Programs; Development and Selection . Occasional Paper No. 

in. Center for Programmed Learning for Business, University of Michigan, Ann 
Arbor, 1965. 24 pp. 

9. Ryan, William F. A Handbook of Programed Learning Information . The State Educa- 

tion Department, Albany, New York, 1964. 59 pp. 

10. Salerno, Albert. An Evaluative Report on the Programed Learning Center . Mahoning 
Valley Vocational School, Experimental Project, Mimeo Report, Vienna, Ohio, 
1965-66, 63 pp. 
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11. Schramm, Wilbur. The Research on Programed Instruction - An Annotated Bibli- 

ography. Office of Education, U. S. Department of Health, Education, and Welfare, 
Bulletin OE-34034, U. S. Government Printing Office, Washington, D. C., 1964. 
114 pp. 

12. Selection and Use of Programed Materials: A Handbook for Teachers . Divisionjjf 

Audiovisual Instructional Service, National Education Association, Washington, D.C., 
1964. 71 pp. 



B. Periodicals and Others 

1. A. V. Communication Review. Department of Audiovisual Instruction, National Educa- 

* “ tion Association, 1201 Sixteenth Street, N.W., Washington, D. C. 20036. 

A quarterly which includes articles and reviews of publications of interest to those 
using programed instruction. Subscription $6.00 per year. 

2. Audiovisual Instruction. Department of Audiovisual Instruction, National Education 

Association, 1201 Sixteenth Street, N.W., Washing^ton, D. C, 20036. 

Published ten times a year; subscription $6.00 per year. Articles provide infor- 
mation regarding educational media materials, techniques, and instructional 
developments. 

3. Bulletin on Programmed Learning . Center for Programmed Learning for Business, 

The University of Michigan, Ann Arbor. 

These bulletins are published regularly and are available upon request. 

4. Bulletin - The Clearinghouse on Self- Instructional Materials for Health Care Facili- 

* ties. The University of Rochester, School of Medicine and Dentistry. 

These bulletins are published regularly and are available to those in the health 
field upon request. 

5. NSPI Journal. National Society for Programmed Instruction, Trinity University, 715 

Stadium Drive, San Antonio, Texas 78212. 

This journal is published monthly; annual subscription $6.00. 

6. Programed Instruction. The Center for Programed Instruction, Institute of Educa- 

tional Technology, Columbia University. (No longer published.) 

Reported research findings and other news about programed instruction. 

7. Training in Business and Industry . Gellert Publishing Corporation, 33 West 60th 

Street, New York, New York 10023. 

Contains up-to-date reports on what techniques business and industry are using, 
as well as advertisements by many producers of programed instruction and other 
educational media. Magazine is published monthly, is free to qualified personnel 
and available by subscription at $5.00 per year to others. 
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Selected List 



Sources of Information 



1. Center for Programmed Learning for Business 
The University of Michigan, 340 South State Street 
Ann Arbor, Michigan 48104 

2. Department of Audiovisual Instruction, National Education Association 
1201 Sixteenth Street, N.W., Washington, D. C. 20036 

3. Educational Media Branch, Office of Education 

U. S. Department of Health, Education and Welfare 
Washington, D. C. 20202 

4. Institute of Educational Technology 
Teachers College 

Columbia University, New York, New York 10027 

5. National Society for Programmed Instruction 
Trinity University 

715 Stadium Drive, San Antonio, Texas 78212 

6. The Clearin^ouse on Self-Instructional Materials for Health Care Facilities 
University of Rochester 

260 Crittenden Boulevard, Rochester, New York 14620 
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APPENDIX C 



CRITERIA FOR ASSESSING 
PROGRAMED INSTRUCTIONAL JMATERIALS 



1962 Interim Report of the Joint Committee 
on Programed Instruction and Teaching Ma- 
chines 

American Educational Research Association 

American Psychological Association 

Department of Audiovisual Instruction, NEA 

Prepared with the cooperation and support 
of the Educational Media Branch, U. S. 
Office of Education, imder NDEA Title VII 



THE members of the AERA-APA-DAVIJoint 
Committee include Harry F, Silberman, 
Evan R. Keislar, Robert Glaser, and Arthur 
A. Lumsdaine, Chairman (American Educa- 
tional Research Association); Richard S. 
Crutchfield, James G. Holland, and Lawrence 
M. Stolurow (American Psychological Asso- 
ciation); and Jack V. Edling, Edward B. Fry, 
Wesley C. Meierhenry, and Paul R. Wendt 
(Department of Audiovisual Instruction, Na- 
tional Education Association). Helpful con- 
tributions were made to the preparation and 
review of the present statement by a cooper- 
ating committee of the American Society of 
Training Directors, whose members are 
Leonard C. Silvern (chairman), Robert L. 
Craig, Stanley L. Levine, Leonard Nadler, 
and Gerald H. Whitlock. Also contributing 
were several consultants and staff assistants, 
including Lloyd O. Brooks, Martin V. Cov- 
ington, H. J. A. Goodman, Bert Y. Kersh, 
Susan M. Markle, Ernst Z, Rothkopf, and 
David G. Ryans. Further suggestions from 
program writers, publishers, or users are 
invited for the committee’s use in the prep- 
aration of subsequent reports. This present 
article, as an interim report, can be in- 
terpreted as expressing a consensus of 
Joint Committee members rather than an 
official policy statement of AERA, APA, and 
DAVI. 



This statement by the AERA-APA-DAVI 
Joint Committee on Programed Instruction 
and Teaching Machines is concerned with 
educational techniques that are variously 
called “programed instruction,” “auto-in- 
structional” methods, and “programmed 
learning.” 

The present statement amplifies and ex- 
tends the previous guidelines published in 
1961 by the Joint Committee.^ This report, 
like the previous one, is addressed pri- 
marily to the non-technical reader inter- 
ested in the possible purchase of programs. 
It summarizes some basic aspects of the 
nature and current status of programed in- 
struction, and also presents some suggestions 
and cautions concerning the assessment of 
programs. 

A subsequent, more technical report will 
provide supplementary information and rec- 
ommendations addressed to the technical 
specialist who is directly concerned with 
obtaining or interpreting quantitative data to 
indicate the effectiveness of programs L. 
contributing to specified instructional goals. 

Programed instruction . As used herein, 
programed instruction refers to the use of 
materials or procedures viiich incorporate 
an “auto- instructional” (or self- instruc- 
tional) program. Such a program commonly 
attempts to provide conditions under which 
a student can learn something efficiently 
with little or no outside help. Current pro- 
grams typically employ a pre-arranged se- 
quence of material that is presented to the 
student one small unit at a time (e.g., a 
sentence or paragraph). Most programs 
require the student to respond actively at 

1. This earlier statement by the Joint 
Committee was published in 1961 in the 
July-August issue of the AV Communication 
Review, the September issue of Audiovisual 
Instruction, and the November issue of the 
14 eA Journal , as well as in a number of 
other educational periodicals. 
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least once for each unit (or **frame^^) of 
material - for example, by composing or 
selecting an answer to a question. Pro- 
grams also commonly provide prompt con- 
firmation or correction, as the case may be, 
for each response the student makes. In 
some cases, the program is presented by a 
mechanism or device called a * ‘teaching 
machine”; in other cases it is presented by 
a specially designed form of book.^ 

Some Basic Considerations Concerning Pro- 
gramed Instruction 

With or without the use of “teaching 
machines” for controlled presentation of 
programs, individual instruction by pro- 
gramed materials offers a very important 
potential resource for education. Attention 
to the following guidelines is suggested, 
however, in order that the potentialities of 
programed instruction may be effectively 
developed and utilized. 

Experimentation and planning for school 
use . Programed instruction represents a 
relatively new and thus far largely experi- 
mental resource for education. Experimental 
tryout in schools, of both locally and com- 
mercially developed programs, is strongly 
encouraged. Wide-scale adoption of any 
particular program may well await the eval- 
uation of one or more provisional tryouts of 
that program. 

Curriculum planning . An important po- 
tential advantage of individual programed 
instruction is that abler learners can proceed 



2. Detailed information about the develop- 
ment of teaching machines and programed- 
instruction concepts up to 1960 is contained 
in the source book. Teaching Machines and 
Programmed Learning, edited by Lumsdaine 
and Glaser and published by DA VI (724 p.; 
$7.50). A briefer treatise is Teaching by 
Machine , by L. M. Stolurow, published in 
1961 by the U.S. Office of Education and ob- 
tainable from the U.S, Government Printing 
Office (173 p.; 65 cents). Reviews of several 
more recent books on programed instruction 
may be found in professional journals such 
as Contemporary Psychology, published by 
the APA, and the AV Communication Review, 
published by DA VI. 



at an accelerated rate throu^ basic course 
material and thereby qualify sooner for ad- 
vanced instruction. On the other hand, suit- 
able programing may enable the slow learner 
to attain higher levels of proficiency than 
would otherwise be possible. Planning for 
adaptation of curricula to accommodate these 
possibilities needs to be undertaken as pro- 
gramed materials of demonstrated quality 
become available. 

Perfecting programs through tryout and 
revision . Programed instruction affords out- 
standing opportunities for perfecting instruc- 
tional sequences through successive revision 
based on detailed records of student response 
to preliminary forms of a program. The 
development of high-quality programs will 
generally entail considerable effort and ex- 
pense. However, if costs can be pro-rated 
over a large number of students, a greater 
research and development effort can be in- 
vested in a program than might otherwise be 
considered feasible. 

Tests of program effectiveness . Although 
the content vhich a program is designed to 
teach may be inferred from careful inspec- 
tion of the program itself, external evidence 
based on student performance is needed to 
demonstrate how well the program actually 
teaches. However, the value of a method of 
instruction can not be tested in the abstract. 
For example, evaluation of a particular text- 
book is not an assessment of the usefulness 
of textbooks in general. A properly conducted 
experimental tryout or field test of a pro- 
gram may provide an assessment of that 
particular program, but does not affordproof 
or disproof ofthevalueof a general “method” 
of programed instruction. 

Experimentation conducted thus far sup- 
ports the expectation that good programs, 
carefully developed, can significantly im- 
prove both the quality and economy of in- 
struction. Whether any particular program 
will do so is subject to question until estab- 
lished by adequate tests of that program. 
Unfortunately, programs may be offered for 
sale that will fall short of the potential value 
of programed instruction - for example, be- 
cause they have not been carefully developed 
through procedures that include sufficient 
tryout and revision to assure their effective- 
ness. 
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‘‘TEACHING MACHINES»» 

Some programs require a machine for 
their presentation, while some are available 
only in book form. Other programs are 
available in two separate versions, one in 
book form and the other for use with a 
machine. In any case, it should be em- 
phasized that so-called teaching machines, 
in themselves, do not teach. Rather, the 
teaching depends on the program of instruc- 
tional materials that may be presented by a 
machine. The comparative merits of ma- 
chine and non-machine presentation of printed 
programs for use in schools is as yet an 
unresolved issue . Any advantage for machine 
over book presentation can not be tested in 
the abstract but would depend on the char- 
acteristics of a particular machine. Some 
machines have demonstrable advantages for 
certain uses, including research; and suitable 
machines are required for programs that 
utilize audio materials. 

Machine characteristics . The variety of 
t 3 Tpes of teaching machines continues to pro- 
liferate, with little standardization. In eval- 
uating any make or model of teaching ma- 
chine, a continuing necessity is thus to assess 
the number and quality of programs avail- 
able for use in it. For some machines the 
user who has sufficient time and skill can 
prepare his own programed materials; for 
other machines this may not be feasible. 
With some machines, a program can be re- 
used indefinitely; for others a new copy of 
the programed material may be required for 
each student. 

For many machines, mechanical depend- 
ability can not yet be taken for granted. As 
with any new mechanical device, potential 
purchasers of teaching machines are well 
advised to seek reliable information on how 
extensively the device has been used in 
schools, what maintenance problems have 
been encountered, and the extent to which 
parts and service are locally available at 
reasonable cost. 

Availability of machines . Existing ma- 
chines differ greatly in complexity and cost; 
prices for most of them range from a few 
dollars to several hundred dollars per ma- 
chine. Any catalog of teaching machines is 
likely to be obsolete as soon as it is printed 
because the field is developing so rapidly. 



New machines appear, and some advertised 
models fail to get into production. Several 
dozen different machines are briefly de- 
scribed and illustrated in a 1962 publication 
by Finn and Perrin.^ A number of these are 
commercially available at present. Others 
have been withdrawn from the market or 
were experimental models that have never 
gotten into production. 

PROGRAMS 

Availability of Programs 

An increasing number of programs is 
becoming commercially available in a variety 
of subject-matter areas. Mere availability 
is no guarantee of quality, however. In 
addition, programs (as well as machines) 
are sometimes announced long before they 
are actually available; also, as noted above, 
some programs are in a format tliat can be 
presented only with a particular kind of 
machine. 

A useful guide to available programs for 
school subjects is a 383-page government 
publication entitled Programs , ^62. ^ This 
publication lists some 120 programs reported 
to be commercially available by September 
1962. These programs span the curriculum 
from elementary to college levels and cover 
a variety of subject matter, including lan- 
guage arts, mathematics, music, physical 
and biological sciences, social studies, and 
business education. The report cited in- 
cludes descriptive information and one or 

3. Finn, J. D., and D. G. Perrin. Teach- 
ing Machines and Programed Learning 1962; 
A Survey of the Industry . Occasional jPaper 
No. 3, NEA Technological Development Proj- 
ect. Washington, D. C.: U.S. Government 
Printing Office, 1962 (Publication No. OE- 
34019; 85 p., 55 cents). See subsequent foot- 
note concerning information presented in 
this report. 

4. The Center for Programed Instruction, 
Inc. Programs, ^62; A Guide to Programed 
Instructional Materials . Washington^ D«C.t 
U.S. Government Printing Office, 1962 (U.S. 
Office of Education, Publication No. OE- 
34015; 383 p., $1.50). See following footnote 
concerning information presented in this 
report. 
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more sample sequences from each program. 
Pertinent data given include the intended 
student population, the number of * ‘frames 
in each program, and its price, but no 
attempt is made to evaluate the programs. ° 
It is anticipated that this compilation will be 
updated by similar guides for subsequent 
years. 

Types of Programs 

Programs are being produced in a variety 
of forms. Thus far they have tended to 
cluster around two or three main types; how- 
ever, new variants or mixtures of types are 
also being introduced. The majority of cur- 
rent programs break the subject matter down 
into a large number of small steps or 
“frames, requiring the student to make one 
or more responses to each step. Such a 
program can be so designed that the student 
will respond to critical aspects of each frame 
or will get practice in performing the specific 
operation that each frame is meant to teach. 
Careful programing requires the programer 
to take great pains to insure that these steps 
embody a logical, well- sequenced progress- 
ion of the subject matter. This applies 
especially to programs that are intended to 
serve as sole or independent sources of 
instruction rather than only as supplements 
to other material. Such programs often pro- 
vide a number of examples to illustrate each 
principle, concept, or act thui is to be learn- 
ed. 

Programs of the kind described above 
are designed to adapt to individual differ- 
ences by allowing each learner to proceed at 
his own rate. In addition, some types of 
programs further adapt by providing for 
“branching^^ to alternate materials. For 
this purpose, frames may include questions 
designed to diagnose the leamer^s needs, 
with directions taking him to alternate mate- 
rial suited to these needs. 

5. The reports on devices and programs 
identified in the two preceding footnotes are 
cited solely for the convenience of the reader, 
and statements concerning them are not to 
be construed as an endorsement by the Joint 
Committee or Its parent organizations, either 
as to completeness or accuracy of the infor- 
mation presented, or of the quality of the 
devices and programs listed in these publica- 
tions. 



In most of the current “branching'^ pro- 
grams, the program is so constructed that 
the choice of a particular answer to a diag- 
nostic question determines which frame will 
be presented next, hicorrect answers may 
take the student to frames containing infor- 
mation designed to correct the error before 
allowing him to continue through the se- 
quence, or to frames that provide supple- 
mental information or practice. 

There is little empirical basis at present 
to favor one general type of program over 
another. It may be anticipated that different 
types of programs will eventually prove to 
be especially useful for particular kinds of 
educational purposes, and that different styles 
of programing may be combined effectively 
in a single program. At the present time, 
however, the general pattern of one type or 
another of programing may be superficially 
followed without necessarily capitalizing fully 
on its potential advantages. 

CRITERIA FOR ASSESSING PROGRAMS 

“IntemaP* and “ExtemaP* Sources Of Infor- 
mation About Programs 

A useful distinction can be made between 
“intemaP^ and “extemaP^ characteristics 
which mi^t serve as possible criteria for 
program evaluation. 

“ IntemaP * characteristics refer to fea- 
tures that can be revealed through visual 
inspection of the program. These include 
both the content of the program and the way 
the program is constructed. Content might 
be described in terms of relative emphasis 
given to various topics as well as jenerel 
organization of the material. Descriptive 
characteristics of program construction 
mi^t include information about the length 
of frames, use of branching sequences, 
techniques of prompting, patterns of repeti- 
tion and review, modes and frequency of 
response called for, procedures and sched- 
uling of reinforcement, and the like. 

“ExtemaP^ information about a program 
refers to features which cannot be observed 
merely by inspecting the program itself, such 
as the way it was developed and character- 
istics of its performance as a teaching in- 
strument. External information of interest 
to a potential purchaser could include such 
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things as the source of program content, 
qualifications of authors, history of the pro- 
gramme development, tryout and revisions, 
and test data indicating gains in achievement 
produced by the use of the program. This 
information, as indicated more fully below, 
may be presented in a descriptive manual 
supplied by the program publisher. 

Critical reviews of programs may furnish 
an additional basis for evaluation. Such re- 
views are beginning to appear in professional 
journals along with reviews of textbooks. 
(Some reviews include data on achievement 
attained by using the program as well as the 
reviewer's opinion about program content and 
style.) 

Pr ograms as Related to Textbooks and to 
Tests 

The applicability of internal and external 
kinds of information as possible criteria for 
evaluating programs may in part be seen by 
comparing programs with textbooks and also 
with educational or psychological tests. 

Programs as compared with textbooks. 
Both programs and textbooks may be in- 
spected to determine what topics are covered 
and the relative amount of attention given to 
each. Such inspection would also indicate 
whether the subject matter is factually cor- 
rect, whether it is current, etc. However, 
despite their similarities, programs differ 
from textbooks in several important respects 
that may affect their evaluation. A program's 
requirement for frequent student response 
generates a special source of data useful 
for revising tiie program in detail. The 
tendency to empirically guided development 
of programs is coupled with an orientation 
toward testing the specific effects produced 
by a program, and toward more sharply 
focused objectives defined in terms of speci- 
fied behavioral outcomes. In addition, the 
program is intended to generate a more pre- 
dictable pattern of student behavior than does 
the study of a textbook, which generally has 
a less specialized purpose in aimingto serve 
as a reference source as well as a sequence 
of instruction. 

Programs compared psychological 
and educational tests . Although programs 
aim primarily to instruct students rather 
than to test them, programs and tests share 



some important attributes. Since both gen- 
erate student-response data as an inherent 
feature, both tend to be developed in terms 
of empirical procedures. The difficulty of 
each item in a program, as in a test, can be 
investigated by presenting the program to 
appropriate samples of students and record- 
ing their responses. Both the program and 
the test have limited ranges of usefulness 
that can be described to the potential user 
in terms of empirical evidence; and in both 
cases it is possible to specify an external 
criterion to indicate the extent to which some 
intended outcome is achieved, as evidenced 
by the kinds of behavior that have been de- 
veloped or differentiated. 

Uses of Internal and External Information 
For Assessing Programs 

Inspecting the subject-matter content of 
progrmns. At the present time, the principal 
recommended use of internal data obtained 
from inspection of the programed materials 
is for determining whether program content 
is appropriate to the educator's objectives. 
As with other educational materials, pro- 
gram titles often are not definitive. Pro- 
grams labeled with the name of a particular 
subject matter can vary widely in terms of 
content and associated instructional objec- 
tives. The prospective purchaser of a pro- 
gram should, therefore, inspect the content 
of the program at least as carefully as he 
would that of a textbook. Preferably he 
should go through the entire program to 
determine what aspects of the subject are 
treated or omitted, and the extent to which 
particular sub-topics are developed. 

Limitations of program inspection . A 
risk in relying on inspection lor evaluating a 
program is that one's perception of its value 
may be inappropriately influenced by his 
reaction to particular structural features of 
the program. For example, certain frames 
or items may seem too difficult or too easy. 
However, the difficulty and appropriateness 
of items in a program, like those in a test, 
generally cannot be judged accurately by 
inspection alone. External data are needed 
- data from an actual tryout of the program 
on students who are representative of the 
population of intended users. 
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The need for test data to assess a pro- 
eram's effectiveness. Empirical evidence on 
what is learned from the program can also 
be a better basis than mere inspection for 
answering such questions as whether program 
sequences have too much or too little repe- 
tition, review, prompting, overlap of steps, 
etc. At present, the scientific evidence is 
not considered sufficient to permit accurate 
prediction in these respects or to justify 
recommendation that adherence to specific 
rules of program construction be used as a 
basis for program evaluation. External evi- 
dence is recommended as the main basis for 
the evaluation of program effectiveness - in 
particular, test data obtained from using a 
program under specified conditions which 
provide dependable measures of gains pro- 
duced in student achievement and of the time 
students require to achieve these gains. 

Uses and assessment of programs . Pro- 
grams may have a variety of uses. For 
example, they may be intended to provide 
the main source from which students are 
expected to learn facts, principles or skills 
- or they may be intended only to review or 
introduce other instruction. Inmost schools, 
programs will probably be used in conjunc- 
tion with other media of instruction. How- 
ever, no matter what eventual use is con- 
templated for a program, it will generally 
help a prospective user to know what the 
program itseli actually contributes to the 
students* knowledge or proficiency - in addi- 
tion to what is contributed by other elements 
in the instructional situation. 

The kinds of effects that can be revealed 
through empirical tryout are limited by the 
content of the achievement tests or other 
measures used to assess these effects. In- 
spection of the program by the prospective 
purchaser, supplemented by independent pro- 

6. This emphasis o external criteria for 
assessing program effectiveness is con- 
sistent with the position earlier advocated in 
Lumsdaine and Glaser*s 1960 ‘‘Concluding 
Remarks** (op. cit., p. 566), and in Roth- 
kopf*s (1961 conference report, “Criteria for 
the Acceptance of Self-Instructional Pro- 
grams,** in Improving the Efficiency and 
Quality of Learning, A. E. Traxler, editor. 
Washington, D. C,: American Council on 
Education, 1962, p. 30-38. 



times suggest additional uses for which a 
program might be suitable, or kinds of pro- 
gram effects which are not indicated by 
field-test data because they were not con- 
templated in the programer*s original pur- 
pose. 

Inspection of achievement-test content. 
Aside from the data obtained in texting a 
program* s use under laboratory or field 
conditions, inspection of the program itself 
as a basis for appraisal can be supplemented 
if the author or publisher has spelled out tbe 
program*s purpose by describing and exhibit- 
ing in full the achievement-test items which 
purport to exemplify what the program is 
intended to teach. These criterion-test 
items, as well as responses called for by the 
program and test, can be examined to see 
what the learner is required to be able to do, 
and whether this reflects the kind of compe- 
tence which the educator wishes to achieve. 
Such an analysis of test content as a basis 
for determining a program* s objectives may 
be particularly helpful for programs which 
are intended to serve as a primary source 
of instruction rather than merely as an 
adjunct to other instructional material. 

REPORTING DESCRIPTIVE INFORMATION 
ABOUT PROGRAMS 

Manila Is for Providing External Data 

“ Manuals** for tests and programs . Be- 
cause some of the characteristics needed to 
appraise educational and psychological tests 
are revealed only throu^ data obtained by 
using them, it has become accepted practice 
to furnish information about test character- 
istics in a manual supplied by the test pro- 
ducer. It appears both desirable and feasible 
to provi'^3 n similar manual for programs 
as a vehicle for presenting relevant external 
information about properties which are not 
apparent on inspection. 

Questions that mig^t be answered about 
a program. Information presented in aman- 
ual can help program producers or dis- 
tributors to answer questions which the 
prospective purchaser may wish to ask as a 
basis for selection. Several areas of such 
questions concerning external information 
about a program may be identified. These 
questions might deal with (1) the program *s 
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purpose and intended use, (2) the source of 
program content, (3) the way the program was 
developed, including tryout and revision, and 
(4) the conduct and results of testing to 
determine empirically the effectiveness, or 
** performance characteristics,^* of the pub- 
lished program. The last of these kinds of 
information will generally be considered the 
most important; however, it also involves 
the kind of data which may be hardest to 
evaluate as to adequacy without specialized 
technical advice. 

Purpose and Scope of a Program Manual 

The kind of manual here suggested could 
apply to all types of programed materials. 
However, some of the details appropriate 
for some programs probably would not apply 
to others. For example, less test data would 
probably be needed in the case of very short 
programs. 

It is expected that the main user of such 
a manual would be the school district or 
other large-scale purchaser interested in 
buying programs in considerable quantity. 
To evaluate fully some of the data that could 
appropriately be included would generally 
require advice from a technical consultant 
who has professional training and competence 
in testing and measurement techniques as 
well as in statistics and experimental design. 
However, the manual also could well supply 
general Interpretive information to help the 
non-technical purchaser determine the pro- 
gramme relevance to his educational purposes. 
Such information could precede and refer to, 
when appropriate, the presentation of the 
technical detail needed for the specialist to 
appraise a program* s effectiveness. 

Program “labels.** In addition, a digest 
of the iirformation in the manual might be 
presented as a brief preface or “label** 
attached to individual copies of the program. 
Such a label could, at a minimum, indicate 
the purpose and intended use of the program, 
who was primarily responsible for its con- 
tent and preparation</ (Wid the source of pub- 
lications in which further data on its de- 
velopment and effectiveness might be found. 
This information should include the age- or 
grade-level(s) of the learners for whom the 
program is designed, and the prerequisite 
skills and abilities these learners are 



assumed to have. The publisher could then 
characterize and briefly illustrate the kind 
of competences the program has been demon- 
strated to produce when used in the manner 
suggested. 

Further Information on Source and 
Development of Program Content 

A more detailed manual which could be 
supplied by the program publisher to pros- 
pective users on request might elaborate 
this minimum information in relation to 
further questions, such as the following, 
which concern the source and development 
of program content. 

Sources of content . What textual or cur- 
riculum sources were used in the selection 
and development of the content? How current 
were these sources? Who were the pro- 
gramer(s) and the collaborating curriculum 
specialists or subject-matter consultants (if 
any), that prepared, edited and reviewed the 
program materials? What are their aca- 
demic and experience qualifications with 
respect to competence in the subject matter 
and techniques of programing? To what kind 
of review was the program material sub- 
jected during its development? 

Development, tryout, and revision . As 
previously noted, records of learners* re- 
sponses to preliminary versions of a pro- 
gram can provide a basis for its progressive 
revision and improvement prior to publica- 
tion. Accordingly, the prospective purchaser 
might wish information about the extent to 
which such tryout and revision has been 
conducted, the kind and amount of student- 
response data obtained, and the way in which 
the data were used in revising the program. 
The manual might also indicate the criteria 
used to determine when the program was 
ready for final release and printing prior to 
the effectiveness testing on which the per- 
formance characteristics of the published 
program are based. As supplementary in- 
formation, the producer might also wish to 
indicate the assumptions made and principles 
used in constructing the program. 

Information About the Demonstrated 
Effectiveness of a Program 

It is to be hoped that the manual for a 
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program, at least for major programs of 
considerable scope, will furnish evidence on 
the programme effectiveness based on meas- 
urement of student performance on pre- and 
post-program criterion tests. These tests 
should be exhibited either in the manual or 
in an available supplement, so as to exemplify 
what the producers expect the student to 
learn as a result of program use. 

Program producers are strongly en- 
couraged to support any claims for the 
effectiveness of the programs in terms of 
gains in student performance produced by 
the final, published version of the program, 
as revealed by appropriate criterion tests. 
A clear distinction should be made between 
this effectiveness-test data for the final 
program and any test data obtained in earlier 
tryouts of preliminary versions used as a 
basis for revision. (Changes made in the 
program after the later effectiveness-test 
data are obtained could throw doubt on the 
validity of these data for a demonstration of 
the programme effectiveness.) 

The manual should present whatever 
further information would seem helpful in 
evaluating the reported effects of the pro- 
gram or the adequacy of the evidence on 
which they are based. It should in all cases 
present evidence to document for the tech- 
nical reader that the gains in achievement 
reported can rightly be attributed to the 
effect of the programme use rather than to 
extraneous causes. In addition, it should 
describe the physical and social conditions 
of the programme use and effectiveness- 
testing procedures in sufficient detail so that 
their essential features could be reproduced 
by another investigator if desired. This 
information would include details of super- 
vision and incentives used, other instruction 
given, size of student groups, and physical 
arrangement of rooms during program use 
and testing. Any material discrepancies 
between recommended conditions of use and 
those that were employed in obtaining the 
effectiveness-test data should be noted. Stu- 
dents’ prior experience with programs and 
teaching machines, if any, should be noted in 
view of spuriously large temporary gains 
that can sometimes result as a novelty effect 
when a new device or procedure is first 
introduced. 

The manual should indicate how many of 



the students started and completed the pro- 
gram, the average length of time they re- 
quired to finish it, the average level of per- 
formance on the specified pre- and post- 
program tests of achievement, and the range 
or variability with respect to these meas- 
ures. Relevant further temporal data would 
include the amount of time learners of differ- 
ent ability spent on various portions of the 
program, how this time was distributed 
(especially for long programs), and the time 
lapse between the completion of the program 
and the criterion test. 

Effectiveness tests could of course be 
conducted so as to include post-program 
measures other than the test that specifies 
the programer’s objectives. The program’s 
effect on secondary objectives not originally 
aimed at could thus also be revealed. How- 
ever, whether or not suchtests are conducted 
by the producer or by others (e.g., by a 
prospective user or by an independent re- 
search agency), it is to the programer’s 
interest to specify what he intended as the 
program’s principal objectives. Finding a 
program to be ineffective or of only limited 
effectiveness for contributing to a secondary 
or unintended objective mi^t be helpful to 
the user in making a decision about the use 
of the program for that purpose, but cannot 
properly be held as a criticism of the pro- 
gramer’s effort. 

It is anticipated that a school district 
contemplating the use of a program will be 
interested in its effect on performances 
other than those tested by the program pro- 
ducer. Particularly in this case, it is 
recommended that, when possible, potential 
users assess a program by their own field 
tests, guided by suitable technical consulta- 
tion, before deciding on adoption of a pro- 
gram for wide-scale use. Performance char- 
acteristics of a program could specify its 
effectiveness in affecting behavior of stu- 
dents describable as changes in knowledge, 
understanding, skill or other outcomes, in- 
cluding beliefs, interests, and motivations. 

Learner characteristics . Specification of 
prior knowledge and ability of learners can 
serve both to identify the pre-program base 
line from which gains may be measured, and 
also to indicate what prerequisites are needed 
in order to learn effectively from the pro- 
gram. Learner characteristics may be 
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specified as an aspect of the program’s 
purpose and intended use. The correspond- 
ing characteristics for the samples of stu- 
dents used in preliminary tryouts or, par- 
ticularly, in the effectiveness -testing of the 
program, should be separately specified so 
as to indicate the degree to which these 
learners were typical or at3rpical of the 
learners for whom the program is intended. 
The producer should also indicate the limits 
(particularly the lower limits) of the popula- 
tion for whom the program is intended, and 
of the samples used in testing its effective- 
ness. 

Technical information concerning the con- 
duct of effectiveness-testing . Valid assess- 
ment of viiat is taught by the use of a pro- 
gram generally involves special technical 
problems. Evaluation should, whenever pos- 
sible, utilize the assistance of technical 



specialists having recognized competence in 
educational measurement and experimental 
design. The analogy of programs with stand- 
ardized educational and psychological tests 
also suggests a precedent for preparation of 
technical recommendations by members of 
relevant professional organizations. These 
recommendations can serve both to help 
insure the technical soundness of effective- 
ness-testing procedures, and to promote 
comparability and interpretability of data by 
fostering consistently high standards of prac- 
tice in reporting the results of tests. The 
further interim report to be published at a 
later date by this Committee ^11 discuss in 
more detail some of the technical problems 
of assessing what the use of a program, in 
and of itself, contributes to definable in- 
structional goals. 



RECOMMENDATIONS FOR REPORTING 
THE EFFECTIVENESS OF PROGRAMED INSTRUCTION MATERIALS 

(Adapted from October, 1965 revision) 



I. INTRODUCTION 



This report has been prepared by the Joint Committee on Programmed Instruction and 
Teaching Machines of the American Educational Research Association, the American Psy- 
chological Association, and Department of Audio-Visual Instruction (National Education 
Association).! The work of this committee has been supported by the Educational Media 
Branch, Office of Education, U. S. Department of Health, Education, & Welfare, under Title 
VII of the National Defense Education Act. The parent associations have charged the com- 
mittee with providing useful guidance to publishers and purchasers of programed instruction 
materials. 2 The present recommendations are intended to help in improving the effectiveness 
of program selection and utilization. This report supplements the 1962-63 committee report, 
to which the reader is referred for background. 3 



Purpose and Content 

The report provides assistance primarily 
to potential users concerned with the selec- 
tion and effective use of instructional pro- 
grams. It also provides guidance to those 
who publish programs or report data on 
program effectiveness. The present report 
deals with recommendations concerning in- 
formation on the effects that a given pro- 
gram can be shown to produce, regardless 



of how these effects may relate to the user’s 
purposes. Supplement I to this report con- 
tains suggestions for information to be in- 
cluded in a program manual for teachers and 
other users who require information about 
program characteristics. Supplement I I con- 
tains recommendations intended to serve as 
a guide for those who are preparing tech- 
nical documentation in support of statements 
about the outcomes that a program can 
produce. 
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n, RECOMMENDATIONS ON REPORTED 
EFFECTIVENESS OF PROGRAMS 

The basic premise of this report is that 
instructional effectiveness must be judged 
for each program, according to its demon- 
strated merits. Evidence for the effective- 
ness of a program should be based on a 
detailed study which has been fully docu- 
mented in a technical report. As empha- 
sized in its 1962-63 report, the committee 
takes the position that effectiveness of each 
program must be determined by measure- 
ment of the instructional outcomes which 
the programme use can be shown to bring 
about. At the present state of the art, users 
generally cannot assess the effectiveness of 
a particular program reliably by mere in- 
spection of the program or by reference to 
statements about its developmental history. 

A. General Recommendations 
Recommendation 1: 

Evidence for the effectiveness of a pro- 
gram should be based on a carefully con- 
ducted study which shows what the programme 
use accomplished under specified conditions. 
Such a study must employ suitable before 
and after measurements, with control pro- 
cedures to insure that effects attributed to 
the program can be clearly distinguished 
from the effects of other instruction. 

Recommendation 2: 

The results of the evaluation study should 
be carefully documented in a technical re- 
port prepared in keeping with accepted stand- 
ards for scientific reporting. (Specific 
recommendations for the preparation of such 
documents are presented in Supplement II to 
this report.) 

Recommendation 3; 

All claims or statements about the effec- 
tiveness of a program should be supported 
by specific reference to the evidence con- 
tained in the technical report. 



B. More Specific Recommendations 

It is assumed that data on the effective- 
ness of programs will be obtained and re- 
ported by: (1) program producers, (2) using 
agencies, including school systems, and (3) 
projects conducted by universities and other 
research agencies. Accordingly, some fur- 
ther specific recommendations and sugges- 
tions, given below, are addressed to pros- 
pective users, program publishers, review- 
ers, and research agencies or institutions 
which conduct or report assessment studies. 

The uses of program-assessment data 
differ depending upon the needs and technical 
experience of the user. For most teachers 
and school administrators, reports are need- 
ed which report the effectiveness of an 
instructional program in fairly straight- 
forward terms that are quickly compre- 
hensible without examining detailed technical 
data. On the other hand, a detailed account 
of all experimental procedures and instru- 
ments used in assessment ^ needed for the 
technical evaluator, who must critically 
analyze the study to see if the summarized 
results and interpretations are warranted. 

1 . Itecommendations for the prospective pur- 

chaser or user: 

a. Prospective users should evaluate each 
program on its own merits according to its 
demonstrated effectiveness in producing 
specified outcomes. 

b. In determining the suitability of any 
program for a particular purpose, the pros- 
pective user should first formulate his own 
objectives in as much detail as possible and 
then evaluate the program in relation to 
these objectives in the li^t of three things: 

(1) The apparent appropriateness of 
the program content for his pur- 
poses, as based on inspection of 
the program itself and of the pro- 
ducer's statement of the programme 
objectives. These objectives may 
be inferred from tests supplied by 
the producer for measuring the 
intended outcomes of the program. 
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(2) Consideration of factors affecting 
practicality , or feasibility of use, 
such as the unit cost of the pro- 
gram, initial and maintenance cost 
of a machine (if required), and 
factors affecting supervision, 
scheduling, and other aspects of 
administration. 

(3) Evidence on the demonstrable 
effectiveness of the program in 
terms of outcomes relevant to the 
user^s objectives. (These may in- 
clude motivational, or attitudinal, 
effects as well as subject matter 
competences.) 

c. The prospective user is advised to 
ignore all claims for the effectiveness of a 
program which are not backed up by appro- 
priate data that have been subjected to com- 
petent evaluation. Advice on tlie soundness 
of claims for program effectiveness should 
preferably be obtained from a technical 
advisor or reviewer who has competence in 
the fields of educational psychology, meas- 
urement, and experimental design and who 
has reviewed available reports on the effects 
of the program in the light of technical 
recommendations identified in Section 5 
herein. 

d. In addition to consulting reviews pub- 
lished in professional journals, users should 
seek all available data on demonstrated 
performance characteristics of the program, 
not only from information supplied by the 
producer but also from reports prepared 
independently: for example, reports pre- 
pared by school systems, research projects, 
or other agencies that have conducted pro- 
gram assessment studies of the particular 
program. 

2. Recommendations for program publishers: 

The following recommendations, plus 
recommendations 4a-4e following, are offer- 
ed to assist the program producer in pro- 
viding necessary information which will help 
users make intelligent choices among avail- 
able programs. ' 

a. The publisher should state in detail 
the minimum objectives of his program, 
preferably in terms of specific behaviors or 
competences which its use is intended to 
achieve for specified kinds of learners , 



b. PulDlishers should refrain from pro- 
moting a program in terms of general state- 
ments about the value of programed instruc- 
tion as a general ‘‘method,^’ or on the basis 
of statements about its effectiveness not 
supported by detailed data, as recommended 
herein. 

c. Publishers should provide a program 
manual, preferably one that can be updated 
or supplemented as new data on the pro- 
gram become available. (See suggestions 
for the content of such a manual provided in 
Supplement 1 to this report.) 

d. Preliminary limited editions of pro- 
grams, prior to validation by definitive 
evaluative studies meeting the conditions of 
technical adequacy indicated herein should 
be issued to facilitate collection of evalua- 
tion data. These should always be clearly 
identified to the purchaser as experimental 
or preliminary editions. 

e. Publishers should use a suitably de- 
scriptive title for the program, one >^ich 
appropriate!)' delimits the scope of the sub- 
ject matter and skills taught. Relatively 
longer titles £ind use of subtitles and detailed 
tables of contents are recommended. 

3, Recommendations for reviewers: 

To assist users in evaluating programs, 
those vfho preptare reviews mi^t, in addition 
to expressing rJieir opinions about the suita- 
bility of the program content and objectives, 
be guided by the following suggestions and by 
recommendations 4a-4d. 

a. Take into account all available assess- 
ment data. 

b. Evaluate and interpret such data in the 
context of technical considerations such as 
those set forth in following sections and 
amplified in Supplement 11 to this report. 

c. Make available example, in a 

supplementary report or by deposit with 
the American Documentation Institute) any 
relevant details of his analysis of assess- 
ment data which require more space than is 
appropriate for a publ.ished review in pro- 
fessional journals. 

d. Utilize a procedure and format of 
reporting which provides a thorou^ anal)rsis 
required of the program. (When appropriate, 
reviewers mi^t consider employing a set of 
topical headings related to program appro- 
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priateness, such as producer's statement 
of program objectives, appropriateness of 
objectives to current curricular concepts, 
suitability of objectives and program to the 
designated student population, etc. Where 
data from a formal study to assess program 
effects are available, the reviewer should 
evaluate the data in the light of the criterion 
measures used, adequacy of description of 
test populations, description of conditions of 
experimental and intended program use, 
degree of correspondence between conditions 
of testing and of intended application, etc.) 

4 . Recommendations addressed jointly to 
program producers, reviewers, and technical 
advisors; 

The following recommendations recognize 
that data on the effects of programs may 
vary from impressions based on observa- 
tions of one or two subjects, as they work 
through a program, to a full-scale, formal 
study in which the program's specific effects 
on learning, retention, motivation, and appli- 
cation of knowledges and skills are de- 
termined for representative populations of 
students under varying conditions of use. 
In the formal study, these data may be 
analyzed to show differing effects for sub- 
groups of varying aptitude and bacP^round. 

Informal tryouts and subjective impres- 
sions can be useful when intended to serve 
as guidance to the programer in revising 
early versions of a program. The teacher 
also receives some value from informal 
tryout when a rough overall “screening test" 
is desired to help decide whether or not a 
program seems to “work" (in the sense of 
being generally suitable for its intended pur- 
pose).^ 

The recommendations that follow are 
concerned primarily with the reporting of 
formal and rigorous assessment studies 
which are required for determining in some 
detail the performance characteristics of a 
program - that is, the specific outcomes 
which a program can be shown to be reliably 
capable of producing. 

a. Reported data on effectiveness should 
refer to the effects produced by the program 
itself, imless other instructional sources are 
clearly identified, and their contribution is 
assessable. (Althou^ most schools use 



programs in conjunction with other media of 
instruction, it will generally help a prospec- 
tive user to know what the program alone 
actually contributes to the student's knowl- 
edge or proficiency, apart from what is 
contributed by other elements in the in- 
structional situation.) 

b. Program producers should cite the 
available evidence from their own studies of 
the program to document any claims they 
make about the effectiveness of the program. 
Publishers as well as reviewers should also 
cite any pertinent evidence available from 
all documented studies of the program that 
are known to them. 

c. Publishers and reviewers should 
differentiate clearly and explicitly between 
(1) mere opinions, of experts or others, 
about the probable effectiveness of the pro- 
gram, and (2) documented evidence on the 
outcomes its use has been actually shown 
to produce. 

d. While brief summary statements may 
often appear in advertising copy or bro- 
chures, or on the cover or label of a pro- 
gram, or in a program manual, or a review, 
such summary statements should always 
reference the technical report on which they 
are based, so that the correspondence be- 
tween interpretive statements and underly- 
ing data is made explicit. 

e. Summaries of information concerning 
a program's demonstrated effectiveness 
should be made widely available in published 
sources of general distribution. It is sug- 
gested that program producers report such 
data in a program manual and that using 
agencies (such as school systems and re- 
search projects) also publish the results 
they obtain, on any commercially available 
program, in appropriate professional jour- 
nals. These agencies should provide copies 
of their reports to the publisher of the pro- 
gram as soon as they are completed and 
ready for publication. 

f. To insure continued availability of 
technical reports and of data not published 
in full in standard books or journals, at 
least one complete copy of the technical 
report and of all basic data tabulations should 
be furnished to a suitable depository such 
as the American Documentation Institute or 
University Microfilms, and this fact should 
be noted in the teacher's manual. 
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5, Recommendations Concerning Technical 
Reporting; 

These recommendations, in addition to 
Recommendations 4a-4f above, are addressed 
to those concerned with obtaining, reporting, 
or evaluating the adequacy of information 
from empirical studies of the effects of 
programs. The recommendations summar- 
ized here are amplified in Supplement II to 
this report.® 

a. In accordance with basic criteria of 
scientific reporting, the entire evaluation 
procedure should be reproducible . This 
applies to the derivation, administration, 
and description of criterion measures as 
well as to the selection of the experimental 
population and all aspects of experimental 
design and procedure. The technical report 
should describe the procedures used and the 
results obtained in such a way that a tech- 
nically-qualified person (1) can assess the 
validity of the statements concerning what 
outcomes the programme use will achieve 
and (2) could replicate the study in substan- 
tially identical fashion. 

b. To satisfy this basic requirement, the 
technical report should give full details on 
all relevant aspects of the evaluation study, 
including criterion measures, character- 
istics of subjects, conditions of program use 
and data collection, experimental design, 
and data obtained. Some of the more specific 
aspects of such topics dealt with in Supple- 
ment n are; 

(1) Procedures employed in measuring 
of retention, transfer, and attitude, 
and other dependent measures such 
as time to reach a criterion. 

(2) Characteristics of students as in- 
dicated by measures of prior knowl- 
edge, competence, intelligence, ap- 
titude, etc. 

(3) Procedures and scheduling of pro- 
gram use, related instruction, con- 
ditions of administration. 

(4) Procedures used in sampling and 
assignment of subjects, use of con- 
trol and comparison groups, con- 
trols for so-called Hawthorne and 
related spurious (in the sense of 
extraneous - Ed. note.) effects, 
and account taken of “palP^ effects. 



(5) Processing, tabulations, analysis, 
imd summarization of data, tests 
of significance and reporting fidu- 
cial limits, etc. 



FOOTNOTES 

1. Committee members are; Harry F. 
Silberman, Evan R. Keislar, Robert Glaser, 
and Arthur A. Lumsdaine, Chairman (AERA); 
Richard S. Crutchfield, James G. Holland, 
and Lawrence M, Stolurow (APA); and Jack 
V. Edling, Edward B. Fry, Wesley C. Meier- 
henry, and Paul R. Wendt (DA VI). Ernst Z. 
Rothkopf served as consultant, and Brett B. 
Hamilton as staff assistant, in the prepara- 
tion of this report. Helpful contributions 
were made to the preparation of the present 
statement by members of a cooperating com- 
mittee of the American Society for Training 
and Development (formerly American Society 
of Training Directors) under the chairman- 
ship of Leonard C. Silvern. The present 
report represents a consensus of the Joint 
Committee members rather than official 
policy of AERA, APA, or DA VI. Further 
suggestions from program writers, publish- 
ers, or users are invited. 

2. A useful guide to available programs 
for school subjects is the government pub- 
lication entitled Programs ^63 , prepared by 
the Center for Programed Instruction (U.S. 
Office of Education. Publication No. OE- 
34015-63; 814 pp., $2.50). This publication 
lists some 350 programs reported to be 
commercially available by the end of 1963, 
and includes descriptive information, price, 
and one or more sample sequences from 
each program, though with no attempt to 
evaluate the programs. Another useful com- 
pilation of programs, which includes more 
programs but ^ves less detail on each pro- 
gram is Programed learning; A Bibliog- 
raphy of Programs and Presentation Device^ 
edited by Carl li, Hendershot, and published 
by Delta College, University Center, Mich- 
igan. This listing has been updated quarterly 
(L/C Cat. No. 64-11824; $3.50). 

3. The 1962-63 report was entitled** Cri- 
teria for Assessing Programmed Instruc- 
tional Materials,” and was published in 
Audiovisual Instruction, February, 1963, pp. 
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84-89; it has also been reprinted in several 
other educational journals and books. 

4. Suggestions concerning such class- 
room tryouts by the teacher are offered in 
the booklet Selection and Use of Programed 
Materials: A Handbook for Teachers , pub- 
lished by the Division of Audiovisual In- 
struction Services of the National Education 
Association, Washington, D. C. (1964; Li- 
brary of Congress catalogue number 64- 
23523); 50^ per copy. 



5. For further background on rationale, 
techniques, and problems in assessing the 
effectiveness of instructional programs the 
reader may wish to consult the chapter by 
A. A. Lumsdaine entitled ‘‘Assessing the 
Effectiveness of Instructional Programs** 
in the book Teaching Machines and Pro- 
gramed Learning II; Data and Directions 
(Ed. by R. Glaser, Washington, D.C.; Na- 
tional Education Association, 1965). 
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